Dropout: Definition and Examples

Dropout is one of the most influential regularization techniques in deep learning, introduced by Geoffrey Hinton and his team in 2012. Its principle is elegantly simple: during each training step, each neuron in the network has a probability p (typically 0.5 for hidden layers and 0.2 for the input layer) of being temporarily 'turned off', meaning its output is set to zero. This forces the network not to rely excessively on any single neuron or small group of neurons.

The intuition behind dropout is that it simulates training an ensemble of different sub-networks at each iteration. Since each neuron can be deactivated at any time, the network learns more robust and distributed representations. Dropout can also be seen as a form of 'structural noise' that prevents the model from memorizing training data instead of extracting generalizable patterns.

In practice, dropout is only applied during the training phase. During inference (when the model makes predictions), all neurons are active, but their weights are multiplied by (1 - p) to compensate for the fact that more neurons are active than during training. This technique, called 'inverted dropout' in its modern variant, performs this compensation directly during training.

Although dropout was initially designed for fully connected neural networks, variants exist for other architectures: spatial dropout for convolutional networks (CNNs), recurrent dropout for recurrent networks (RNNs/LSTMs), and DropConnect which deactivates connections rather than neurons. In modern Transformer architectures like GPT or BERT, dropout is still used on attention layers and feed-forward layers.