How would you quantify uncertainty in model performance when outcome events are scarce?

Why

With 5,000 patients and 20 events, sensitivity is estimated from 20 people. A model catching 18 has sensitivity 0.90, but the Wilson 95 percent interval runs from about 0.70 to 0.97, and the simple normal approximation gives an upper limit above 1 that must be truncated. Riley and colleagues stress that small validation samples give imprecise and potentially misleading estimates, and that commonly cited minimums for c statistic and calibration slope are at least 100 events and 100 non-events.

The same logic applies to every metric. AUC depends on how many positive and negative pairs exist, calibration curves need events spread across the risk range, and a subgroup with five events cannot support any claim. Intervals make that limitation visible to the reader instead of hiding it behind a point estimate.

How to work it through

1. Count events and non-events in each evaluation set and subgroup before computing anything.
2. Use exact or score-based intervals, such as Wilson or Clopper-Pearson, for proportions like sensitivity and PPV.
3. Use bootstrap resampling, at the patient level, for AUC, calibration slope and net benefit, repeating the full evaluation in each resample.
4. Report every estimate with its interval and the number of events behind it.
5. Avoid subgroup claims where events are very few, or pool information with a model rather than splitting.
6. If intervals are too wide for the decision, plan to collect more events rather than over-interpret.

Where the answer stops holding

Bootstrap intervals can be unstable with very few events and may undercover. Intervals reflect sampling noise, not bias from leakage or a non-representative cohort. Repeated looks at a small validation set to choose among models also inflate apparent precision.

Bayesian approaches can help but depend on the prior chosen.