thebench.bio #1 Ecosystem for the Bio-Tech Industry
All Statistics questions

Question 5 of 9 Advanced Concept

What is the difference between statistical significance and practical importance?

Short answer

Statistical significance addresses whether data are difficult to reconcile with a specified null model. Practical importance asks whether the effect is large and meaningful enough to change a real decision. A tiny effect can be statistically significant in a huge dataset and still be clinically irrelevant.

Why

Start by separating evidence of an effect from its consequence. A p-value summarizes how incompatible the observations are with a specified statistical model. It does not tell us the probability that the hypothesis is true or how valuable the effect is.

Suppose a treatment improves a symptom score by only 0.2 points on a 100-point scale. With enough participants, a narrow confidence interval might exclude zero. Yet the difference could fall far below any patient-relevant threshold.

Conversely, a smaller study may yield an estimate with potential clinical importance but a wide interval that includes no effect. Report the absolute and relative effect when appropriate, uncertainty, adverse effects, cost and the minimally important difference defined for the use case. Ideally define decision thresholds before reading results.

A failure to cross p = 0.05 is not proof of no effect, and a p-value below that threshold is not proof of clinical value.

Common mistakes

  • Claiming that p < 0.05 proves a treatment works in a meaningful way or that p > 0.05 proves equivalence