How would you detect silent changes in data type or missing-value encoding in a scientific pipeline?

Short answer

Declare the expected schema and missing-value tokens when reading, instead of letting pandas infer them. Then assert dtypes, null counts and value ranges at each stage boundary, so a change fails the run rather than flowing into results.

Why

read_csv guesses types from the text it sees. One stray "<LOD" in a concentration column turns the whole column into text, and a mean that once worked now raises or, worse, a later coercion turns every value it cannot parse into NaN. Missing-value tokens are guessed too.

By default pandas treats strings such as NA, NULL, None and an empty field as missing, so a column where NA is a legitimate code, for example a sodium flag, loses real values. A vendor who starts exporting -999 or 0 for no reading creates the opposite problem, because those parse as valid numbers. With low_memory left on, pandas also reads large files in chunks and can infer different types for the same column in different chunks.

None of these raise an error. Stating dtype and na_values explicitly, with keep_default_na=False where codes matter, turns guessing into a contract you can test.

How to work it through

Write down the expected schema for each input: column names, dtypes, allowed missing tokens and plausible ranges.
Read files with explicit dtype, na_values and keep_default_na arguments rather than inference.
Immediately after reading, assert that the dtypes match the schema and that no column became object or str unexpectedly.
Log the null count and the count of sentinel values such as -999 or 0 for each column and compare them with the previous run.
Fail the pipeline when a count moves beyond an agreed limit rather than only printing a warning.
Keep a small reference file with known edge cases in the test suite so a library upgrade that changes parsing is caught before production.

An example

An ELISA export normally has 96 numeric concentrations per plate with about 3 missing wells. After a software update, the reader writes below-range wells as "<0.1". With inference, the column becomes text and a later pd.to_numeric(errors="coerce") turns 21 wells into NaN.

The plate mean rises because the low values vanished. With an explicit float dtype, read_csv raises on the first "<0.1" instead. The team then maps "<0.1" to a censored flag in a separate column, and the null-count check shows 3 missing wells, matching the history.

Where the answer stops holding

Schema checks catch type and encoding drift, but not a value that is valid in type and range yet wrong in meaning, such as a unit change from ng to µg in the same column. Tight limits on null counts can fail on genuinely unusual but correct runs, so thresholds need review. Validation libraries help at scale, but the decisions about what counts as missing still come from the assay owner.

A strong answer covers

Declare dtypes and missing tokens instead of inferring them
Default NA strings can swallow legitimate codes
Numeric sentinels such as -999 parse as real values
Assert dtypes and null counts at stage boundaries
Fail the run on drift, keep a reference edge-case file

Common mistakes

Using pd.to_numeric with errors set to coerce as a cleaning step, which silently converts every unexpected token to NaN
Relying on default missing-value parsing when a column contains the literal code NA as valid data
Checking only that the pipeline finished, not that column types and null counts stayed the same

The interviewer might ask next

How would you represent a value below the limit of detection without losing the information that it was measured?