Question 2 of 6 Fundamentals Concept

What could cause unstable training loss in a deep model?

Short answer

Unstable loss can reflect optimization settings, poor input scaling, faulty labels, exploding gradients, batch composition or numerical instability. Diagnose the failure before treating it.

Why

Reproduce the loss spikes on a fixed batch and inspect input distributions, target alignment, optimizer steps, gradient norms and whether any value becomes NaN or infinite. A learning rate that is too high can cause oscillation, while rare extreme batches or mislabeled samples may cause occasional spikes. In a sequencing classifier, mixing normalized and raw counts across batches can produce drastic feature-scale changes.

Plot loss and gradient statistics by step and layer, then change one factor at a time, beginning with preprocessing and learning-rate tests. Gradient clipping may bound unusually large updates but will not fix corrupt labels, leaking targets or data from a shifted measurement process. Record software and data versions, random seeds and hardware.

A smooth training curve does not prove that the model predicts meaningful biology.

Common mistakes

Increasing clipping strength until the loss curve looks smooth while leaving data errors or split leakage unresolved

The interviewer might ask next

Which measurement would distinguish a learning-rate problem from malformed input batches?