What is data leakage in model evaluation?

Why

Leakage is a mismatch between what the evaluation allows the model to see and what real use would permit. A common example is fitting a normalization transform on the full dataset before creating train and test splits. That lets the evaluation set affect the learned transform.

Other forms arise when repeated samples from a patient appear in different folds, when features encode post-outcome events or when labels are indirectly available through metadata. The correct workflow starts by defining the prediction time and independent unit, then separating groups or time periods before learning preprocessing, feature selection or tuning hyperparameters. Pipelines help keep fitted transformations inside each training fold.

An untouched final test set and external validation are valuable when available. High validation performance does not prove absence of leakage, but unexpectedly large gains warrant careful examination of data provenance and split logic.