How would you interpret attention maps when assessing whether a model learned meaningful biology?

Why

It is tempting to look at which residues of a protein or which regions of a slide a transformer attends to and call those the important biology. Jain and Wallace tested this across many NLP tasks and found attention weights were often uncorrelated with gradient-based importance, and that very different attention distributions could produce equivalent predictions. They concluded standard attention should not be treated as explanation.

Wiegreffe and Pinter replied that the answer depends on how explanation is defined, and proposed tests such as comparing with a uniform-attention baseline and checking variance across random seeds. The practical reading for a scientist is that attention is one view inside a model with many layers and heads, and information also flows through residual paths that attention maps do not show. To use it responsibly, check whether attention is stable across seeds, compare with a baseline, and perturb the attended region to see whether the prediction actually changes.

Then compare the highlighted positions with independent knowledge, such as known binding sites or a pathologist's annotation. Agreement with prior biology is encouraging but can be circular, because people notice the maps that match what they expected. A model attending to slide edges or scanner marks is a useful warning even if the map proves nothing else.