Question 12 of 20 Intermediate Troubleshooting

Training loss decreases while validation loss oscillates. How would you investigate?

Short answer

First check whether the swings are just noise from a small or unbalanced validation set. Then look at the learning rate, since steps that are too large make the model jump between solutions, and at differences between training and validation preprocessing or modes. Smooth evaluation and a learning rate schedule usually settle it.

Why

Validation loss is an estimate from a finite set. With 60 validation images, one or two confidently wrong predictions move the mean cross-entropy a lot, so some oscillation is expected. If the swings are much larger than resampling the validation set would produce, something else is at work.

A learning rate that is too high makes the weights bounce around the region of good solutions. Training loss still trends down because it is averaged over many updates, while validation, measured at single points, swings. The Deep Learning book's practical chapter calls the learning rate perhaps the most important hyperparameter to tune, and a decaying schedule or ReduceLROnPlateau in PyTorch often removes the swings.

Mismatches are the next suspect: augmentation applied to validation data, normalization statistics computed differently, or forgetting model.eval(), which leaves dropout active. Batch normalization with small batches can also make evaluation unstable. Watch the trend too.

Oscillation around a rising average is overfitting, not noise, and calls for early stopping or regularization. Finally, compare accuracy with loss, because loss is sensitive to overconfident errors while accuracy may be steady.

A strong answer covers

Estimate noise from validation set size
Too-high learning rate causes bouncing
Schedules or ReduceLROnPlateau
Check eval mode and preprocessing consistency
Rising average means overfitting, not noise

Common mistakes

Selecting the checkpoint at the single lowest validation loss when swings are mostly noise
Applying random training augmentation to the validation set
Reading oscillating loss as instability while accuracy is steady and loss reflects a few overconfident errors

The interviewer might ask next

How would you choose a checkpoint when validation loss is noisy?