thebench.bio #1 Ecosystem for the Bio-Tech Industry
All Model Evaluation questions

Question 9 of 20 Fundamentals Concept

What does k-fold cross-validation estimate, and why would you stratify the folds?

Short answer

K-fold cross-validation splits data into k parts, trains on k minus 1 and tests on the remaining one in turn, and averages the scores to estimate how the modeling procedure performs on new data from the same population. Stratifying keeps each fold's class proportions close to the whole dataset's, which matters when a class is rare.

Why

A single train and test split wastes data and depends on luck. With k-fold, every sample is tested exactly once, and the spread across folds shows how much the estimate varies. The scikit-learn guide describes StratifiedKFold as returning folds that each contain approximately the same percentage of each class as the full set.

With 200 samples and 20 positives, plain 5-fold could give one fold 1 positive and another 8, making fold scores erratic or undefined for metrics such as AUC. Stratification gives each fold about 4. Two points are often misunderstood.

First, cross-validation estimates the performance of the procedure, including preprocessing and tuning, not of one particular fitted model. Second, it estimates performance on new samples exchangeable with the training data. If samples cluster by patient, site or time, random folds put related samples on both sides and overestimate performance, so use group-aware splitters such as GroupKFold or StratifiedGroupKFold, or time-ordered splits.

Fold scores are not independent, because training sets overlap, so the standard deviation across folds understates the true uncertainty. Repeating with different shuffles gives a more stable mean.

A strong answer covers

  • Each sample tested once, scores averaged
  • Estimates the procedure, not one fitted model
  • Stratification keeps class balance per fold
  • Group or time splits when samples are related
  • Fold scores are correlated, spread understates uncertainty

Common mistakes

  • Using plain KFold on a rare outcome so some folds have almost no positives
  • Treating the standard deviation across folds as a confidence interval for future performance