thebench.bio #1 Ecosystem for the Bio-Tech Industry
All Machine Learning questions

Question 19 of 20 Advanced Troubleshooting

A classifier trained on a small, high-dimensional omics dataset reports near-perfect cross-validated accuracy. What would you check?

Short answer

Check first whether feature selection, normalization or tuning used all samples before cross-validation, which creates selection bias. Then check for duplicated or related samples across folds and for labels confounded with batch. Rerun with every step inside the folds and with permuted labels.

Why

With 60 samples and 20,000 genes, some genes will separate the classes by chance alone. Ambroise and McLachlan showed in PNAS in 2002 that published near-zero error rates for gene-expression classifiers were biased because genes were selected using all the samples, so cross-validation was not external to the selection. They recommended performing cross-validation or the bootstrap external to selection, and 10-fold rather than leave-one-out.

The scikit-learn pitfalls page reproduces the effect: selecting features on all data gives accuracy far above chance even with random labels.

How to work it through

  1. Read the code and list every step that learns from data, including filtering, normalization, batch correction, feature selection and tuning.
  2. Move all of them inside a Pipeline evaluated by cross-validation, nesting the tuning.
  3. Check that replicates, repeat biopsies or related samples from one patient stay in one fold by using a group splitter.
  4. Tabulate class against batch, site and processing date to see whether labels are confounded with technical factors.
  5. Run a permutation test that shuffles labels and repeats the entire pipeline, which should give chance-level accuracy.
  6. Confirm the final result on an independent cohort if one exists.

An example

A model separates responders from non-responders in 48 tumors with 98 percent leave-one-out accuracy using 30 genes. The 30 genes were picked by t-test on all 48 samples. Moving selection inside 10-fold cross-validation drops accuracy to 71 percent.

A table then shows 20 of 24 responders were sequenced in batch 1, and within batch 1 alone accuracy is 58 percent. With labels permuted the full pipeline gives 52 percent. The team reports the confounding and requests a balanced validation set.

Where the answer stops holding

A pipeline with every step inside the folds removes selection bias but not confounding with batch, which needs design or batch-aware validation. Permutation tests show whether performance beats chance, not whether it generalizes to new sites. With very few samples, even a correct estimate has wide uncertainty, so report intervals.

A strong answer covers

  • Selection on all samples biases the estimate
  • Every data-dependent step inside the folds
  • Group splitting for related samples
  • Check confounding of labels with batch
  • Permutation test of the full pipeline

Common mistakes

  • Cross-validating only the classifier after choosing genes on the full dataset
  • Reading high accuracy on a tiny cohort as proof the signature is biological
  • Running a permutation test that permutes labels after feature selection rather than before