thebench.bio #1 Ecosystem for the Bio-Tech Industry
All Machine Learning questions

Question 18 of 20 Advanced Practical

Why do you need nested cross-validation when tuning hyperparameters?

Short answer

Choosing hyperparameters by the best cross-validation score uses those folds to fit the choice, so the same score is optimistic as a performance estimate. Nested cross-validation tunes inside an inner loop and estimates performance on outer folds the tuning never saw.

Why

The scikit-learn nested cross-validation example states it plainly: choosing parameters that maximize non-nested CV biases the model to the dataset and gives an overly optimistic score, and the size of the effect depends on dataset size and model stability. With a small dataset and a large grid, the best of 200 configurations is partly the luckiest. When there is enough data, a single held-out test set does the same job.

With 100 to 300 samples, as is common in translational studies, a held-out set is small and noisy, so nested CV uses all the data while keeping tuning and evaluation apart.

How to work it through

  1. Define the full modeling pipeline, including preprocessing and feature selection, as one estimator.
  2. Wrap it in GridSearchCV or RandomizedSearchCV with an inner cross-validation splitter.
  3. Pass that search object to cross_val_score or cross_validate with an outer splitter, using group-aware splitters if samples share donors.
  4. Report the mean and spread of the outer scores as the performance estimate.
  5. Note how much the chosen hyperparameters vary across outer folds, as a sign of stability.
  6. Finally, refit the search on all data to obtain the model you deploy, and describe the outer score as the estimate for that procedure.

An example

A team has 160 tumor samples and tunes an SVM over 48 combinations. The best inner score from plain 5-fold grid search is AUC 0.86. With nested CV, 5 outer folds by 4 inner folds, the outer AUCs are 0.79, 0.74, 0.81, 0.70 and 0.77, a mean of 0.76.

Three folds choose C equal to 1 and two choose C equal to 10. The team reports 0.76 with its spread, refits the search on all 160 samples for the final model and notes the 0.10 gap as the optimism the plain search would have reported.

Where the answer stops holding

Nested CV estimates the performance of the whole tuning procedure, not of one fixed final model. It multiplies computation by the number of outer folds. It does not fix leakage from duplicated donors or batch structure, so splitters must respect groups at both levels.

And it estimates performance within the same population, not at a new site.

A strong answer covers

  • Tuning on the same folds makes the score optimistic
  • Inner loop tunes, outer loop evaluates
  • Whole pipeline inside the inner loop
  • Report outer mean and spread, note hyperparameter stability
  • Refit on all data for deployment

Common mistakes

  • Reporting best_score_ from GridSearchCV as the expected performance
  • Doing feature selection once on all data and then running nested CV only over the model
  • Thinking nested CV produces a single final model with that exact score