Encoder Decoder: Definition and Examples

The Encoder-Decoder architecture is a fundamental paradigm in artificial intelligence, particularly in natural language processing and computer vision. It relies on a simple yet powerful principle: decomposing a complex task into two distinct steps. The encoder reads and analyzes the input (text, image, audio signal) to condense it into a dense vector representation, often called latent vector or context.

The decoder then takes this compressed representation and progressively generates the desired output, element by element. In machine translation, for instance, the encoder reads the source sentence in French and produces a vector capturing its meaning, then the decoder generates the English translation word by word from that vector.

This architecture underwent a major evolution with the introduction of the attention mechanism by Bahdanau et al. in 2014, and then the Transformer by Vaswani et al. in 2017. The Transformer replaced recurrent networks (RNN/LSTM) with self-attention mechanisms, enabling parallel processing and better long-distance dependency capture. Models like T5, BART, and mBART use this full encoder-decoder architecture.

It is important to distinguish this architecture from models that use only one of its components. Models like BERT use only the encoder (ideal for understanding), while GPT and Claude use only the decoder (ideal for generation). The full encoder-decoder architecture excels in transduction tasks, where one sequence must be transformed into another sequence of a different nature.