Question 3 of 6 Intermediate Scenario
When does AUROC mislead in a low-prevalence use case?
Short answer
AUROC can look impressive even when most positive predictions are false alarms in a rare-outcome setting. It summarizes ranking across thresholds but does not directly show how many alerts will be correct at the prevalence and decision threshold that matter in practice.
Why
AUROC describes how often a randomly chosen positive instance receives a higher score than a randomly chosen negative one. That is useful for discrimination, but it does not tell us the practical predictive value of the alerts a clinical workflow will act on. Suppose a condition occurs in only 1% of patients.
Even with a very good false-positive rate, the large number of unaffected patients can produce more false alarms than true cases. Precision-recall analysis makes this trade-off visible, while sensitivity, specificity, predictive values and workload at a chosen threshold make it operational. AUROC is not wrong or useless.
It answers a different question. To evaluate deployment, also examine calibration, confidence intervals, relevant subgroups and whether the validation data represent the target population.
An example
A screening tool ranks cases well but triggers 200 alerts for every 1,000 patients, of which only a small fraction are real positives. AUROC alone does not show the clinical workload.
Common mistakes
- Saying AUROC is invalid whenever classes are imbalanced instead of explaining the missing operational information
The interviewer might ask next
Which metrics would you report if a clinician could review only ten alerts per day?