A validation cohort has much higher prevalence than the deployment population. What would you revisit?

Why

Suppose a sepsis model was validated in an ICU cohort where 30 percent had the outcome, and will run on general wards where 3 percent do. With sensitivity 0.85 and specificity 0.90, PPV is about 0.79 at 30 percent prevalence but about 0.21 at 3 percent, so four of every five alerts would be false. Van Calster and colleagues note that a model developed where the event is common may systematically overestimate risk where it is rarer, which shows up as a negative calibration intercept, meaning predictions run high on average.

The threshold chosen to balance alerts in the ICU will flood the wards. What to revisit is therefore concrete. Recompute predictive values and expected alerts per day at deployment prevalence.

Check calibration-in-the-large on local data and, if needed, update the intercept, which shifts all predictions without changing their ranking. Re-choose the threshold with the people who will act on alerts, ideally with decision curve analysis at relevant risk thresholds. Also check whether the case mix differs, not just the rate, because ward patients with sepsis may present differently, which would change sensitivity and specificity too.

A short silent deployment, scoring patients without alerting, gives local data for all of these checks.