thebench.bio #1 Ecosystem for the Bio-Tech Industry
All Model Evaluation questions

Question 19 of 20 Advanced Scenario

What evidence would you require before concluding a model improvement is meaningful rather than noise?

Short answer

A paired comparison on the same splits and seeds showing a difference larger than its uncertainty, tested with a method that accounts for overlapping training sets, and large enough to matter for the decision. Ideally the gain also holds on an independent dataset and in the subgroups you care about.

Why

A new model scoring AUC 0.81 against 0.79 may simply have drawn luckier splits. Cross-validation folds share training data, so their scores are correlated and a naive paired t-test is overconfident. The scikit-learn example on statistical comparison of models explains this and implements the Nadeau and Bengio corrected t-test, in frequentist and Bayesian forms.

Bouthillier and colleagues add that variance from data sampling, initialization and hyperparameter choice must be included for a comparison to be trustworthy.

How to work it through

  1. Agree in advance on the metric, the minimum difference that would matter and the evaluation data.
  2. Run both models on identical repeated cross-validation splits with several seeds and the same tuning budget.
  3. Compute the paired difference for each split and test it with a corrected resampled t-test, or report a Bayesian probability that the improvement exceeds the minimum.
  4. Check that the gain holds in calibration and decision metrics, not just AUC.
  5. Look at key subgroups to make sure the average gain does not hide losses.
  6. Confirm on an independent test set or external cohort before switching models.

An example

Over 10 repeats of 5-fold cross-validation, a new model's AUC averages 0.812 and the old one's 0.798, a mean paired difference of 0.014. The naive paired t-test gives p well below 0.001, but the corrected test gives p of about 0.09. The team had set 0.02 as the smallest useful gain.

On a later external cohort of 1,200 patients the difference is 0.006 with an interval spanning zero, so they keep the simpler model.

Where the answer stops holding

Corrected tests reduce but do not remove the dependence problem. Statistical significance does not mean the improvement is worth the cost of change. Results on one dataset may not transfer.

Repeatedly comparing many candidates against the same data needs a correction for multiple comparisons or a fresh test set.

A strong answer covers

  • Paired comparison on identical splits and seeds
  • Corrected resampled t-test for overlapping folds
  • Prespecified minimum meaningful difference
  • Check calibration, utility and subgroups
  • Confirm on independent data

Common mistakes

  • Using a standard paired t-test on cross-validation folds
  • Calling any statistically significant gain meaningful without a prespecified minimum
  • Comparing a tuned new model with an untuned old one