thebench.bio #1 Ecosystem for the Bio-Tech Industry
All Machine Learning questions

Question 7 of 9 Intermediate Troubleshooting

How would you prevent leakage when a dataset includes multiple scans from each patient?

Short answer

Split by patient, not by scan. All scans, derived features and duplicates belonging to one patient must remain within the same train, validation or test partition.

Why

Random scan-level splitting leaks person-specific signatures, scanner artifacts and correlated pathology across partitions. The apparent performance can reflect recognition of patients rather than generalization to unseen patients. Preprocessing fitted before the split introduces a separate form of leakage.

How to work it through

  1. Build a stable patient identifier before any sampling.
  2. Reserve an untouched test population, then use GroupKFold or similar patient-grouped cross-validation for model selection.
  3. Fit imputers, scalers, feature selection and harmonization inside each training fold.
  4. Audit repeated studies, near-identical images, sites and patients scanned at multiple hospitals.
  5. For generalization, add a separate external hospital or prospective temporal evaluation.

An example

A model receives CT slices from the same patient on both sides of a random split and reports AUROC 0.97. After patient-grouped splitting, it drops to 0.76. The first figure did not estimate performance for new patients.

Where the answer stops holding

Family members, copied imaging studies and shared acquisition sessions may still create dependence across patient groups. If the use case is predicting future scans of known patients, formulate a different deployment-aligned split explicitly.

A strong answer covers

  • Split by patient, not by scan
  • All scans, derived features and duplicates belonging to one patient must remain within the same train, validation or test partition
  • Build a stable patient identifier before any sampling

Common mistakes

  • Splitting after augmentation
  • Using patient ID as a feature
  • Hyperparameter tuning against the final test group