thebench.bio #1 Ecosystem for the Bio-Tech Industry
All Model Evaluation questions

Question 17 of 20 Advanced Practical

How would you design an external validation study for a clinical risk prediction model?

Short answer

Test the frozen model, unchanged, on data from a different setting or period that matches the intended use. Size the study to estimate calibration, discrimination and clinical utility precisely, define outcomes and predictors as in deployment, and report following TRIPOD+AI.

Why

Internal validation shows how a model performs on data like its training set. External validation asks whether it still works where it will be used, with different patients, clinicians, equipment and time. Riley and colleagues note that many external validation studies are too small to be reliable.

Rules of thumb suggest at least 100 events and 100 non-events for the c statistic and calibration slope, and 200 of each for calibration curves, but they recommend calculating sample size for the precision needed in the specific setting. TRIPOD+AI, a 27-item checklist that replaced TRIPOD 2015, sets out what to report.

How to work it through

  1. Define the intended use, population, setting and decision the model supports, and choose a cohort that represents it, ideally from new sites and a later period.
  2. Freeze the model, its preprocessing and threshold, and record versions before seeing outcomes.
  3. Calculate the sample size needed for target confidence interval widths for calibration, the c statistic and net benefit.
  4. Measure predictors as they would be available at prediction time and ascertain outcomes blind to the predictions.
  5. Report calibration with intercept, slope and a curve, discrimination with intervals, decision curves and performance in key subgroups.
  6. Report any recalibration separately from the frozen-model results.

An example

A model predicting postoperative delirium was developed in two teaching hospitals. The external study uses four community hospitals from the following year. With an expected outcome rate of 12 percent, the calculation for a calibration slope interval of about 0.2 wide gives roughly 1,700 patients, about 200 events.

The frozen model shows c statistic 0.76, against 0.81 at development, a calibration intercept of -0.35 meaning overestimation, and slope 0.88. Updating the intercept corrects mean calibration, and net benefit is positive between 10 and 30 percent thresholds.

Where the answer stops holding

One external cohort shows transportability to that setting only, so performance can still vary between sites. Retrospective data may record predictors differently from live use. Outcomes defined by codes can be noisy.

A model that fails external validation may still be useful after local updating, which then needs its own evaluation.

A strong answer covers

  • Frozen model on a new setting or period
  • Cohort matches intended use
  • Sample size from target precision, not only rules of thumb
  • Calibration, discrimination and net benefit with intervals
  • Report per TRIPOD+AI, recalibration separately

Common mistakes

  • Retuning the model on the external data and reporting it as external validation
  • Running external validation with 40 events and drawing firm conclusions
  • Reporting only the c statistic and omitting calibration