Transformer: Definition and Examples

The Transformer is a deep neural network architecture introduced in the seminal paper "Attention Is All You Need" (Vaswani et al., 2017). Unlike recurrent architectures (RNN, LSTM) that processed sequences word by word, the Transformer processes an entire sequence in parallel thanks to a mechanism called "self-attention." This innovation enabled massive gains in training speed and ability to capture relationships between distant words in a text.

The core of the Transformer rests on three key components: embeddings (vector representations of tokens), the multi-head attention mechanism (which allows the model to "look" simultaneously at different parts of the input sequence), and feed-forward layers. The original architecture has an encoder (which understands the input) and a decoder (which generates the output), but many variants use only one of the two — GPT and Claude use only the decoder, while BERT uses only the encoder.

What makes the Transformer revolutionary is its ability to scale. By increasing the number of parameters, the size of training data, and computational power, performance improves predictably — this is known as "scaling laws." It is this property that has led to the race for ever larger models, from GPT-2 (1.5 billion parameters) to GPT-4 and Claude, which have hundreds of billions.

Today, the Transformer is no longer limited to text. This architecture has been successfully adapted to vision (Vision Transformer / ViT), audio, video, robotics, and even molecular biology (AlphaFold). It has become the universal foundation of modern generative artificial intelligence.