Layer Normalization: Definition and Examples

Layer Normalization is a method introduced by Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton in 2016. Unlike Batch Normalization, which computes its statistics over the entire batch for each neuron, Layer Normalization operates over all neurons in a given layer for each individual example. This fundamental distinction makes it independent of batch size.<br><br>Concretely, for each input, the technique computes the mean and standard deviation of all activations within a given layer, then normalizes these values to obtain a centered and reduced distribution. Two learnable parameters — a scale factor (gamma) and a shift (beta) — are then applied to allow the network to retrieve the optimal representation if needed.<br><br>Layer Normalization has become an essential component of the Transformer architecture, used in every attention block and every feed-forward sublayer. It stabilizes training by reducing the problem of internal covariate shift and accelerates model convergence. Without it, training large language models like GPT or BERT would be considerably more unstable.<br><br>In modern architectures, two main variants are distinguished: Post-Layer Normalization (applied after the sublayer, as in the original Transformer) and Pre-Layer Normalization (applied before the sublayer, adopted by GPT-2 and subsequent models). The Pre-LN variant improves training stability and allows removing learning rate warm-up in many cases.