thebench.bio #1 Ecosystem for the Bio-Tech Industry
All Machine Learning questions

Question 16 of 20 Intermediate Interpretation

What does a learning curve tell you about whether more data would help?

Short answer

A learning curve plots training and validation scores against training set size. If both scores are low and have converged, the model is limited by bias and more data will not help much. If training is high, validation lower and the gap still closing, the model is limited by variance and more samples are likely to help.

Why

Collecting more samples in biology is expensive, so it is worth asking before you ask for budget. The scikit-learn guide describes the learning curve as showing training and validation scores for increasing numbers of training samples, to find out how much more data would help and whether error comes mainly from variance or bias. Its example shows a naive Bayes model whose two curves converge at a low score, so more data would not help, and an SVM whose training score stays well above its validation score, so more data probably would.

Read the shape, not one point. Suppose a classifier trained on 50, 100, 200 and 400 patients reaches validation AUC of 0.66, 0.71, 0.74 and 0.76 while training AUC sits near 0.95. The gap is large and validation is still rising, so 800 patients might plausibly reach about 0.78, though the gains are shrinking.

If instead both curves had flattened at 0.70, a richer feature set or a more flexible model would be the next step, not more patients. Each point should be averaged over several splits with its spread shown, because small subsets are noisy. A learning curve also assumes new data come from the same population.

Adding patients from a new site is a different question about transfer.

A strong answer covers

  • Scores plotted against training set size
  • Converged low curves mean bias, more data helps little
  • Large closing gap means variance, more data helps
  • Average over splits and show spread
  • Assumes new data from the same population

Common mistakes

  • Promising a performance level from extrapolating a curve far beyond the data observed
  • Reading a single small-subset point that is mostly noise as a trend