thebench.bio #1 Ecosystem for the Bio-Tech Industry
All Deep Learning questions

Question 17 of 20 Advanced Practical

How would you design an ablation experiment to find which part of a multimodal model adds value?

Short answer

Train and evaluate matched variants, each removing one modality or component, on identical splits with several random seeds and the same tuning budget. Compare the differences with their spread across seeds and folds, and count a component as useful only if removing it reliably hurts held-out performance.

Why

A model combining histology images, expression data and clinical variables may score well, but the images might add nothing beyond age and stage. An ablation answers that by removing parts one at a time. The trap is noise.

Bouthillier and colleagues showed that variance from data sampling, weight initialization and hyperparameter choice markedly affects benchmark results, so a single run per variant can mislead. A fair ablation holds everything else fixed and measures that variance. It also gives each variant a fair chance.

A full model tuned for weeks compared with ablated versions run once on its settings will always look better, and the difference reflects effort, not the modality. Removing a component is an intervention on the model, so the conclusion is about this architecture and this data, and is strongest when the same pattern holds across folds and seeds.

How to work it through

  1. Define the comparisons in advance: full model, each single modality, and the full model minus each modality or fusion component.
  2. Fix the data splits, grouped by patient, and use the same splits for every variant.
  3. Give every variant the same hyperparameter search budget, so the full model is not the only one that was tuned.
  4. Run each variant with several seeds, for example five seeds by five folds.
  5. Report the mean and spread of the difference from the full model for each variant, using paired comparisons on the same folds.
  6. Add a strong simple baseline, such as logistic regression on clinical variables, to show what the deep model adds over it.

An example

A survival model uses whole-slide images, RNA-seq and clinical data for 900 patients. Across 5 folds and 5 seeds, the full model reaches a concordance index of 0.71, with a standard deviation of 0.02. Clinical only reaches 0.66, and clinical plus RNA reaches 0.70.

Removing images costs 0.01 on average, smaller than the seed-to-seed spread, while removing RNA costs 0.04 in every fold. The team reports that RNA adds clear value and the image branch does not yet, and drops it from the next iteration to save compute.

Where the answer stops holding

Removing a modality changes the model's capacity and training dynamics, so an ablation shows the value of a component in this architecture, not in general. Interactions mean the effect of removing two parts is not the sum of removing each. Results apply to the population and splits used.

Missing modalities at deployment time are a separate question, needing tests with modalities dropped at inference.

A strong answer covers

  • Prespecified matched variants
  • Same patient-grouped splits for all
  • Equal tuning budget for every variant
  • Multiple seeds and paired comparisons
  • Simple baseline for context

Common mistakes

  • Tuning the full model carefully and running ablated variants with its hyperparameters unchanged
  • Drawing conclusions from one seed per variant
  • Interpreting the drop from removing one component as its effect in any architecture