Question 4 of 6 Intermediate Scenario
How would you diagnose exploding or vanishing gradients?
Short answer
Log gradient norms by layer and step, compare early and late layers, and inspect activations, learning rate and update sizes. Exploding gradients cause extreme updates. Vanishing gradients impede learning upstream.
Why
For a deep encoder, register diagnostics around backward passes and track gradient norm distributions across layers alongside loss and parameter update ratios. Very large, suddenly rising norms and nonfinite updates suggest explosion. near-zero gradients in early layers despite a learning signal can indicate vanishing. Check activation saturation, initialization, normalization layers, architecture depth, sequence length and accidental detach operations.
Confirm that gradient accumulation and mixed-precision scaling are handled correctly before comparing raw values. Perform a small-batch overfit test and vary one condition to isolate causes. Clipping can mitigate explosion, while residual connections, normalization and architecture or initialization changes may improve signal propagation.
A small gradient is not always a bug: converged parameters and intentionally frozen layers can have near-zero updates.
Common mistakes
- Diagnosing gradients solely from a noisy loss chart or confusing frozen parameters with vanishing gradients
The interviewer might ask next
What additional measurement would reveal an accidental detach in a feature extractor?