Why is a softmax output not automatically a calibrated probability?

Why

A classifier outputs logits, and softmax converts them into a distribution. Training with cross-entropy pushes the correct class up but, especially once training accuracy is near perfect, keeps rewarding larger margins, so outputs drift toward 0 and 1. Guo and colleagues found that modern neural networks, unlike those of a decade earlier, are poorly calibrated, and identified depth, width, weight decay and batch normalization as factors.

They also found a simple fix effective on most datasets: temperature scaling, a single-parameter variant of Platt scaling that divides logits by a constant learned on a validation set. It changes confidence but not the ranking of classes, so accuracy is unchanged. This matters when a probability drives a decision.

If a pathology model says 0.95 for malignancy, a clinician needs that to mean roughly 95 in 100. Check it with a reliability diagram that bins predictions by confidence and compares with observed accuracy, and with a proper scoring rule such as the Brier score or log loss. Calibration is also local to a population.

A model calibrated at one hospital can be miscalibrated at another with different prevalence or scanners, so recalibrate on local data and recheck after any shift.