thebench.bio #1 Ecosystem for the Bio-Tech Industry
All Model Evaluation questions

Question 14 of 20 Intermediate Practical

How would you recalibrate a classifier, and which data would you fit the calibrator on?

Short answer

Fit a monotone mapping from the model's scores to observed outcome rates, using sigmoid or Platt scaling for small datasets and isotonic regression when there are more than about a thousand samples. Fit it on data the model was not trained on, and judge the result on yet another set.

Why

Calibration adjusts predicted probabilities so that among cases given 0.2, about 20 percent have the outcome. scikit-learn's CalibratedClassifierCV fits either a sigmoid, based on Platt's logistic model, or isotonic regression. Its documentation notes that isotonic is more flexible but prone to overfitting on small datasets, and performs as well as or better than sigmoid with more than about 1,000 samples. It also says the user must ensure the data used to fit the classifier is disjoint from the data used to fit the calibrator.

Because the mapping is monotone, ranking metrics such as AUC are generally unchanged.

How to work it through

  1. Measure calibration first on held-out data with a reliability curve, the Brier score and the calibration intercept and slope.
  2. Decide whether a fix is needed and whether it is global or specific to a new site.
  3. Reserve a calibration set not used for training, or use CalibratedClassifierCV with internal cross-validation.
  4. Choose sigmoid for small calibration sets and isotonic when there are enough samples.
  5. Fit the calibrator, then evaluate calibration and discrimination on a separate test set.
  6. Document the calibration data, method and date, because recalibration may need repeating when the population changes.

An example

A gradient-boosted model predicts 90-day readmission. On 2,500 held-out patients, mean predicted risk is 21 percent but the observed rate is 14 percent, with calibration slope 0.78. Using 1,500 of them as a calibration set, isotonic regression is fitted.

On the remaining 1,000, mean predicted risk becomes 14.5 percent, the slope 0.97 and the Brier score falls from 0.121 to 0.109, while AUC stays at 0.74.

Where the answer stops holding

Recalibration fixes the mapping from scores to probabilities, not poor ranking. A calibrator fitted at one site may not suit another, so local recalibration is often needed. Isotonic regression produces step functions that can give identical probabilities to many patients.

Calibration in rare subgroups needs larger samples than overall calibration.

A strong answer covers

  • Monotone mapping from scores to observed rates
  • Sigmoid for small data, isotonic for larger
  • Calibrator fitted on data disjoint from training
  • Evaluate on a separate test set
  • Ranking metrics largely unchanged

Common mistakes

  • Fitting the calibrator on the same data the model was trained on
  • Using isotonic regression on a few hundred samples and reporting calibration on those same samples
  • Expecting recalibration to raise AUC