Mixture of experts
Model design in which a trainable gate selects a combination of experts for each input; in the sparse form only a fraction of the parameters is active for each token.
In plain language
A gate chooses which parts of the network handle an input.
In a sentence
- The mixture of experts model selected a subset of its experts for the input.
- We distinguished total parameters from active parameters in the mixture of experts model.
Usage note
The sparse form activates a subset of the network for an input. Total parameter count and active parameter count therefore describe different quantities.
See also
- Transformer
Neural network architecture built around attention mechanisms that relate positions in a sequence without relying on recurrent or convolutional layers for that relationship.
- Scaling laws
Measured relationships in which language model loss follows a power law as model size, training dataset size and the compute used for training change.
Cite this term.
CC BY 4.0, attribution required.
Orbyt Labs, Mixture of experts (AI Glossary), Nov 14, 2026. https://
Orbyt Labs Glossaries (Nov 14, 2026), https://
Last updated .