AI Glossary

Mixture of experts

noun phraseNeural networkspreferred in this glossary

Model design in which a trainable gate selects a combination of experts for each input; in the sparse form only a fraction of the parameters is active for each token.

Source: Shazeer et al. 2017 (opens in a new tab) (adapted)

Adapted: adds the fraction of parameters activated for a token from Meta's description of mixture of experts models. Source: Meta AI, Llama 4 2025 (opens in a new tab)

In plain language

A gate chooses which parts of the network handle an input.

In a sentence

  • The mixture of experts model selected a subset of its experts for the input.
  • We distinguished total parameters from active parameters in the mixture of experts model.

Examples written for this glossary

Usage note

The sparse form activates a subset of the network for an input. Total parameter count and active parameter count therefore describe different quantities.

See also

  • Transformer

    Neural network architecture built around attention mechanisms that relate positions in a sequence without relying on recurrent or convolutional layers for that relationship.

  • Scaling laws

    Measured relationships in which language model loss follows a power law as model size, training dataset size and the compute used for training change.

Cite this term.

CC BY 4.0, attribution required.

This term

Orbyt Labs, Mixture of experts (AI Glossary), Nov 14, 2026. https://www.orbytlabs.ai/orbyt-collective/glossary/mixture-of-experts

The dataset (JSON)

Orbyt Labs Glossaries (Nov 14, 2026), https://www.orbytlabs.ai/glossary#dataset

Last updated .