Question 6 of 6 Advanced Concept
Why do documented exclusions matter when interpreting an experiment?
Short answer
Exclusions change the analyzed sample and can bias effects. Prespecified, transparent criteria help distinguish quality control from outcome-driven selection.
Why
Before collecting or analyzing outcomes, define measurable exclusion conditions such as sample authentication failure, prespecified contamination thresholds or missing critical measurements. Track every excluded unit, its reason and the timing of the decision. Compare group counts and relevant covariates before and after exclusions and include a sensitivity analysis retaining questionable observations where scientifically appropriate.
A decision to discard a control sample because it weakens significance changes the inference and damages trust. At the same time, keeping a genuinely contaminated sample can also mislead. Explain the effect of exclusions on the estimand and generalizability, and publish a transparent flow diagram or log.
A documented exclusion is not automatically justified. Readers need to judge whether the rule and its application were independent of desired results.
Common mistakes
- Removing inconvenient observations without a consistent rule or recording how sample selection changed
The interviewer might ask next
Which sensitivity check would show whether your conclusion depends on two excluded observations?