thebench.bio #1 Ecosystem for the Bio-Tech Industry
All Deep Learning questions

Question 11 of 20 Intermediate Troubleshooting

How would you distinguish underfitting from optimization failure in a neural network?

Short answer

Try to overfit a tiny subset, even a single batch. A working model and training loop should drive training loss close to zero on it. If it cannot, the problem is optimization or a bug. If it can, but full-data training loss stays high, the model lacks capacity or the features lack signal.

Why

High training loss has two very different causes. Underfitting means the model, as designed, cannot represent the pattern. Optimization failure means it could, but training never finds good weights, because of a bad learning rate, poor initialization, a wrong loss or a bug.

The Deep Learning book's practical chapter notes that even small models can usually fit a sufficiently small dataset, so failing to do so points to a defect. Karpathy's widely used training recipe makes the same check a routine step: overfit one batch of a few examples and confirm the loss reaches its minimum.

How to work it through

  1. Check the loss at initialization against its expected value, for example about the log of the number of classes for balanced cross-entropy.
  2. Train on one batch of 2 to 16 examples with regularization off and confirm the loss approaches zero.
  3. If it does not, inspect labels, the loss function, input normalization and the learning rate, and check that parameters are actually changing.
  4. Once one batch fits, train on the full set and compare with a simple baseline such as logistic regression on the same features.
  5. If training loss stays high while the tiny set fit easily, increase capacity or improve the inputs.
  6. Track gradient norms per layer throughout to spot vanishing or exploding gradients.

An example

A CNN classifying four cell phenotypes plateaus at training loss 1.38, close to the log of 4, which is about 1.386, so it is predicting uniformly. On a batch of eight images it still cannot get below 1.3. Inspection shows the labels were one-hot encoded and passed to CrossEntropyLoss alongside a softmax layer that was applied twice.

After removing the extra softmax, the eight images fit to near zero loss, and full training reaches 0.45 with 81 percent validation accuracy.

Where the answer stops holding

Overfitting a tiny batch proves the pipeline can learn, not that it generalizes. Some architectures with batch normalization behave oddly on very small batches, so use a handful of examples rather than one. A model can fit training data and still underfit a harder validation distribution, which is a different problem.

A strong answer covers

  • Overfit a single small batch as a test
  • Check initial loss against its expected value
  • Failure on tiny data means bug or optimization issue
  • Compare with a simple baseline on full data
  • Monitor gradient norms by layer

Common mistakes

  • Adding layers to fix high training loss before checking the pipeline can fit eight examples
  • Applying softmax before a loss that already expects raw logits
  • Keeping dropout and weight decay on while testing whether a tiny batch can be memorized