Question 2 of 6 Intermediate Scenario
How would you test whether an analytical result survives reasonable methodological choices?
Short answer
Predefine plausible alternative analyses, rerun them transparently and report where conclusions are stable or sensitive, rather than choosing only favorable results.
Why
Start with the key estimand or scientific question and document primary model assumptions. Identify defensible alternatives such as normalization strategies, covariate sets, quality thresholds or missing-data handling. If possible, preregister the analysis family and specify a clear metric of stability.
Execute all variants on the same well-described dataset and compare direction, magnitude, uncertainty and threshold dependence of results. For example, an association that changes sign when a single outlier is removed deserves more caution than one with similar effect size across reasonable specifications. A multiverse or specification curve can make such sensitivity visible, but unplanned exploratory variants must be labeled as exploratory.
Do not automatically discard sensitive findings, and do not treat agreement between many correlated analyses as independent replication.
Common mistakes
- Reporting only the preferred model and describing its p-value as proof of robustness
The interviewer might ask next
What would make a sensitivity analysis informative rather than a search for significance?