How do batch size and learning rate interact during training?

Why

Each step estimates the true gradient from a minibatch. The Deep Learning book notes that the standard error of that estimate falls only with the square root of batch size, so 100 times more examples per batch costs 100 times more computation but cuts the noise only tenfold. Small batches add noise that can act as a regularizer, and the book notes they may need a small learning rate to stay stable.

The learning rate, which the same book's practical chapter calls perhaps the most important hyperparameter, scales each update. Too high and the loss jumps or diverges, too low and training crawls or stalls. Goyal and colleagues trained ResNet-50 with batches up to 8,192 images without loss of accuracy by scaling the learning rate linearly with batch size and adding a warmup period at the start.

That linear scaling rule is a useful starting point, not a law, and it broke down beyond their largest batches. In practice, memory often fixes the batch size, especially for 3D images, so you tune the learning rate for it. Gradient accumulation can simulate a larger batch, but batch normalization statistics still come from the small physical batch.

When reporting results, state both values.