Question 8 of 9 Intermediate Troubleshooting
Model performance falls sharply after deployment to another hospital. How would you investigate?
Short answer
A performance collapse at another hospital is a distribution-shift investigation, not automatic evidence that the model algorithm is defective.
Why
Case mix, event prevalence, outcome definitions, assay or scanner protocols, missingness and coding practices can all shift. Discrimination, calibration and operational utility may deteriorate for different reasons. Data pipeline bugs can masquerade as genuine domain shift.
How to work it through
- Verify data and labels first: cohort inclusion, time windows, units, censoring, outcome adjudication and feature availability at prediction time.
- Compare distributions and missingness by site, check calibration plots and subgroup performance, and quantify uncertainty.
- Separate covariate, label and concept shift hypotheses.
- Seek a temporally or externally held-out evaluation before retraining.
- Recalibrate only when its assumptions and clinical governance justify it.
An example
The external hospital has lower disease prevalence and uses a different troponin assay. Precision falls and predicted risks are high even while AUROC changes little. Fixing units and assessing recalibration matters more than adding another network layer.
Where the answer stops holding
Small subgroup samples produce unstable metrics. A local recalibration can hide label-quality problems. A shifted intervention policy may change the outcome mechanism itself, requiring a broader rethink of the target.
A strong answer covers
- A performance collapse at another hospital is a distribution-shift investigation, not automatic evidence that the model algorithm is defective
- Verify data and labels first: cohort inclusion, time windows, units, censoring, outcome adjudication and feature availability at prediction time
Common mistakes
- Reporting only AUROC. fine-tuning on external test labels and then calling the result external validation
- Ignoring workflow context
The interviewer might ask next
When might recalibration be appropriate but full model retraining inappropriate?