Question 6 of 9 Advanced Scenario
How would you test whether a model relies on batch artefacts?
Short answer
Examine performance under batch-held-out evaluation, quantify batch predictability from features and assess whether important predictors correlate with acquisition or processing metadata.
Why
First map sample collections, plate positions, sequencing runs, reagent lots, sites and labels. If cases were mainly processed on one plate and controls on another, batch information is inseparable from the target without new data. Train a nuisance classifier to predict batch from the features and inspect whether leading features encode instrument signatures.
Evaluate the primary model when whole batches are withheld, and compare with donor-held-out results. Apply batch correction only using training data and without destroying genuine biological variation. Confirm with orthogonal assays or a prospectively balanced experiment.
Feature importance alone cannot prove causal reliance because correlated variables may substitute for each other. Report confounding explicitly when no clean counterfactual evaluation is possible.
Common mistakes
- Removing batch effects on the entire cohort before a held-out test and claiming leakage-free validation
The interviewer might ask next
What new experiment would you design if every disease case came from one sequencing run?