thebench.bio #1 Ecosystem for the Bio-Tech Industry
All Deep Learning questions

Question 18 of 20 Advanced Scenario

What risks arise when fine-tuning a foundation model on a small clinical cohort?

Short answer

With few patients, full fine-tuning can memorize the cohort, distort useful pretrained features and give unstable results across seeds. Validation becomes noisy, leakage between pretraining data and the test set is hard to rule out, and the model may not transfer to other sites.

Why

Foundation models are pretrained on large unlabeled or loosely labeled data, so a cohort of 150 patients seems enough to adapt them. The risks come from the mismatch between millions of parameters and a few hundred labels. Full fine-tuning can overfit quickly, and Kumar and colleagues showed it can also distort pretrained features so that it underperforms a simple linear probe under distribution shift, while training the head first and then fine-tuning combined the benefits.

Lighter options, such as a linear probe on frozen embeddings or parameter-efficient adapters, should be compared before full fine-tuning. Evaluation is the second risk. A test set of 30 patients gives very wide confidence intervals, and repeated tuning against it turns it into a training signal, so nested or repeated cross-validation grouped by patient is safer.

A third risk is overlap. If the foundation model was pretrained on public data that include your test images or patients, results are inflated, so check the documented pretraining sources. Fourth, small single-site cohorts carry site-specific artifacts, so external validation matters.

Finally, the training data must be permitted for this use, and model cards and data provenance help a reviewer judge all of this.

A strong answer covers

  • Overfitting and feature distortion with full fine-tuning
  • Compare linear probe, adapters and staged fine-tuning
  • Noisy evaluation on a small test set, use grouped resampling
  • Possible overlap with pretraining data
  • Single-site artifacts need external validation

Common mistakes

  • Fine-tuning all parameters first instead of comparing against a frozen-feature baseline
  • Reporting performance on a 30-patient test set without confidence intervals
  • Ignoring whether the foundation model's pretraining data overlap the evaluation data