thebench.bio #1 Ecosystem for the Bio-Tech Industry
All Python for Scientific Computing questions

Question 18 of 20 Advanced Practical

How would you design a small test suite for an analysis that transforms raw instrument signals into summary statistics?

Short answer

Test each transformation on tiny inputs whose answers you can work out by hand, add edge cases such as empty, saturated and missing signals, check invariants that must always hold, and keep one end-to-end regression test on a real reference run with stated tolerances.

Why

A signal pipeline usually has a few stages: baseline subtraction, normalization, peak or well calling, then summary statistics. Most bugs live at the edges of those stages, such as an off-by-one window, a divide by zero on a flat trace or an aggregation that skips NaN. Small hand-checkable cases catch logic errors.

Invariants catch whole classes of error, for example that subtracting a constant baseline never changes peak positions. A regression test on real data catches drift from library upgrades. Floating-point outputs need tolerances, so numpy.testing.assert_allclose is the right assertion, with rtol and atol chosen from the instrument's precision rather than left at defaults.

How to work it through

  1. List the pipeline stages and write the expected behavior of each in one sentence.
  2. For each stage, write two or three pytest cases with tiny arrays whose results you can compute by hand, using parametrize to cover several inputs.
  3. Add edge cases: an empty trace, a constant trace, a saturated detector value and a missing value.
  4. Write invariant tests that must hold for any input, optionally with Hypothesis generating random inputs.
  5. Freeze one real instrument run and its approved outputs as a regression fixture and compare with assert_allclose at agreed tolerances.
  6. Run the suite in continuous integration on every change and on every dependency upgrade.

An example

A fluorescence plate-reader script subtracts the mean of blank wells, then reports per-condition mean and coefficient of variation. A unit test feeds four wells with values 110, 120, 130, 140 and blanks of 10, and expects a mean of 115 and a CV of 11.2 percent using the sample standard deviation. An edge test with all blanks missing must raise a clear error rather than return NaN.

An invariant test checks that adding 1,000 to every well and blank leaves the CV unchanged. The regression fixture is one 96-well run, compared at rtol 1e-6.

Where the answer stops holding

A green suite shows the code does what the tests say, not that the method is scientifically valid for a new assay. Regression fixtures freeze past behavior, including past bugs, so approve them deliberately. Tests that are too tight fail on harmless numerical changes and teach people to ignore failures.

Coverage numbers do not show whether the important edge cases were chosen.

A strong answer covers

  • Hand-checkable unit tests for each stage
  • Edge cases: empty, constant, saturated, missing
  • Invariants that hold for any input
  • Regression fixture from a real approved run
  • Tolerances from instrument precision via assert_allclose

Common mistakes

  • Testing only the happy path with a typical plate, so flat or saturated traces are never exercised
  • Comparing floating-point outputs with exact equality and then loosening tolerances until tests pass
  • Regenerating the regression fixture whenever it fails without reviewing why the output changed