What do SHAP values explain, and what can they not tell you?

Why

Lundberg and Lee introduced SHAP in 2017 as a unified framework that assigns each feature an importance value for a particular prediction, with theory showing a unique solution with desirable properties. For a patient whose predicted risk is 0.32 against an average of 0.10, SHAP might attribute 0.12 to high troponin, 0.07 to age and 0.03 to the rest, summing to the difference. Summing absolute values across patients gives a global ranking.

That is useful for checking whether a model uses sensible inputs, and for spotting a model leaning on a site code or scanner identifier. The limits are as important. SHAP describes the model, so a model that learned an artifact gets a faithful explanation of the artifact.

The SHAP documentation has a whole article warning against reading predictive models as causal, because a feature can carry credit for a confounder it is correlated with. Correlated features also share or swap credit depending on how the explainer handles feature dependence, so two reasonable settings can rank them differently. Exact computation is costly, so most explainers approximate, and explanations of a poorly calibrated or overfit model inherit its faults.

Treat SHAP as a diagnostic that raises hypotheses, then test them with designed experiments or held-out data.