Question 9 of 9 Advanced Practical
Design an analysis plan that separates confirmatory hypotheses from exploratory comparisons.
Short answer
A confirmatory analysis tests a prespecified decision claim with controlled error rates. Exploratory analyses discover hypotheses and should be labeled as such even when their p-values are small.
Why
Selecting endpoints, subgroups or thresholds after looking at outcomes increases false-positive opportunities and effect-size exaggeration. Multiplicity control matters across the relevant family of inferential claims, not necessarily every descriptive analysis.
How to work it through
- Before data review register the primary hypothesis, estimand, outcome, population, planned contrast, sample-size rationale, missing-data handling and multiplicity strategy.
- Clearly list secondary confirmatory and exploratory analyses.
- Lock an analysis plan with version history and preserve blinded review where practical.
- Report effect sizes, intervals and negative results for all prespecified comparisons.
- Label deviations, their timing and rationale.
An example
A study prespecifies one efficacy endpoint but later identifies a striking treatment effect in one of 24 post-hoc subgroups. That result can motivate a validation study, but should not be presented as the original confirmatory objective.
Where the answer stops holding
Adaptive designs can modify procedures according to preplanned rules. This is not the same as unblinded outcome fishing. Genuine hypothesis-generating work need not suppress exploratory findings.
It must describe the uncertainty honestly.
A strong answer covers
- A confirmatory analysis tests a prespecified decision claim with controlled error rates
- Exploratory analyses discover hypotheses and should be labeled as such even when their p-values are small
- Before data review register the primary hypothesis, estimand, outcome, population, planned contrast, sample-size rationale, missing-data handling and multiplicity strategy
Common mistakes
- Calling all p<0.05 findings confirmatory
- Using post-hoc power to justify the claim
- Silently changing the primary endpoint
The interviewer might ask next
How would you document an unavoidable amendment after partial unblinding?