Question 8 of 20 Fundamentals Concept

Why do neural networks need nonlinear activation functions, and why is ReLU a common default?

Short answer

Without a nonlinearity, any stack of linear layers collapses into a single linear function, so depth adds nothing. ReLU, max(0, z), is the usual default for hidden layers because it is cheap and does not saturate for positive inputs, which keeps gradients from shrinking.

Why

A linear layer multiplies by a matrix and adds a bias. Two linear layers in a row are just another matrix and bias, so a hundred of them can only draw straight-line decision boundaries, like logistic regression. Putting a nonlinear function after each layer lets the network bend and combine features, which is what allows it to model interactions such as a drug working only when two genes are both expressed.

The Deep Learning book states that in modern networks the default recommendation is the rectified linear unit, g(z) = max(0, z). It also explains why older choices fell out of favor for hidden layers. Functions that saturate, becoming very flat, make gradients very small, and sigmoid and tanh saturate for large inputs.

ReLU has a constant gradient of one for positive inputs, so signals pass back through many layers. It has its own failure: a unit whose input is always negative outputs zero and gets zero gradient, a dead unit, often caused by a learning rate that is too high. Variants such as leaky ReLU and GELU address this, and transformers commonly use GELU.

Output layers are different, because they must match the task: softmax for classes, sigmoid for independent labels, none for regression.

A strong answer covers

Stacked linear layers collapse to one linear map
Nonlinearity enables interactions and curved boundaries
ReLU does not saturate for positive inputs
Sigmoid and tanh saturate and shrink gradients
Dead ReLU units and variants such as GELU

Common mistakes

Believing that adding more linear layers increases what a network without activations can represent
Using ReLU on the output layer of a regression that must predict negative values

The interviewer might ask next

How would you detect that many ReLU units in a layer have died during training?