Why might a model with better discrimination produce worse clinical decisions at a chosen threshold?

Why

AUC rewards ordering patients correctly. A clinical decision, such as whether to biopsy, applies a cutoff to the predicted risk, and the right cutoff reflects how many unnecessary biopsies are worth one cancer found. Vickers and colleagues explain decision curve analysis, which computes net benefit across such threshold probabilities and compares a model with treating everyone or no one.

Net benefit is sensitivity times prevalence minus the false positive rate times one minus prevalence, weighted by the odds at the threshold, and its unit is true positives per patient. Take 1,000 patients with 10 percent prevalence and a threshold of 20 percent, where the weight is 0.25. Model A has the higher AUC but overestimates risk, so at that threshold it flags patients with sensitivity 0.95 and specificity 0.55, and its net benefit is about -0.006, worse than treating nobody.

Model B has lower AUC but is calibrated, giving sensitivity 0.70 and specificity 0.85 and net benefit about 0.036, or 36 extra true positives per 1,000 with no excess harm. Recalibrating Model A might reverse the result, which is the point: discrimination is necessary but not sufficient. Decision curves depend on the prevalence and threshold range, so show the range clinicians would actually use.