thebench.bio #1 Ecosystem for the Bio-Tech Industry
All Model Evaluation questions

Question 4 of 6 Intermediate Scenario

How are discrimination and calibration different?

Short answer

Discrimination describes how well a model ranks people with different outcomes. Calibration asks whether predicted probabilities match observed event frequencies. A model can rank patients well while systematically exaggerating their risk.

Why

Imagine a model that always gives higher predicted risk to patients who later experience an event than to patients who do not. It may have high discrimination, measured by metrics such as AUROC. But if it predicts 80% risk for patients whose actual event rate is closer to 20%, its probability estimates are badly calibrated.

Calibration can be examined with reliability plots, observed-to-expected summaries, calibration intercept and slope, and proper scoring rules such as the Brier score. Calibration must be evaluated in the population and time period where the model will be used because event prevalence and case mix can shift. Recalibration can sometimes improve probability accuracy without changing ranking, but it must be fitted using appropriate validation data.

In clinical applications, both properties matter because decisions depend on risks, not just ordering.

An example

A risk model identifies which patients are higher risk, but the hospital discovers that its quoted 30% risk group experiences the outcome only 10% of the time.

Common mistakes

  • Using accuracy or AUROC as evidence that probabilities are well calibrated