Question 2 of 6 Advanced Practical
How do you evaluate a model on an imbalanced biomedical dataset?
Short answer
For an imbalanced biomedical classifier, I would report more than accuracy or AUROC. I would evaluate sensitivity, specificity, precision or predictive value, precision-recall performance, calibration and uncertainty at a threshold aligned with the intended clinical use.
Why
Start with the decision the model will support and how common the target condition is. Accuracy can be misleading if a model simply predicts the majority class. AUROC describes ranking but may conceal a flood of false positive alerts when the event is rare.
Precision-recall curves and positive predictive value show what proportion of flagged cases may be real at the evaluated prevalence. Calibration answers whether numerical risk estimates correspond to observed frequencies. Report confusion-matrix counts and uncertainty intervals, assess clinically important subgroups, and avoid tuning the final threshold on the same test set used for reporting.
Use grouped validation where repeated biological samples or patients could leak between folds. A model with impressive aggregate metrics can still be unusable if it misses critical cases or overwhelms clinicians with alerts.
A strong answer covers
- Avoid relying only on accuracy or one threshold