Mixture Of Experts: Definition and Examples

Mixture of Experts (MoE) is a machine learning architecture in which a model is composed of multiple independent sub-networks, called "experts", each specializing in processing certain types of data or tasks. A key component, the routing network (or "gating network"), dynamically decides which experts to activate for each input. Thus, only a fraction of the total model is invoked at each inference.

The main advantage of this approach is decoupling total model size (number of parameters) from actual computational cost. An MoE model can contain hundreds of billions of parameters but activate only tens of billions per request. This makes it possible to achieve the performance of a very large dense model while maintaining inference speed comparable to a much smaller model.

In the context of large language models (LLMs), the MoE architecture was popularized by models like Mixtral from Mistral AI or GPT-4 from OpenAI (whose presumed architecture is based on MoE). Specifically, each transformer layer contains multiple feed-forward experts, and the router typically selects the 2 most relevant experts out of 8 or 16 available to process each token.

For the end user, MoE has a direct impact: it makes more powerful, faster, and cheaper models possible. It is one of the architectural innovations that explains why recent models are both more powerful and more accessible than their predecessors.