AI Glossary

Distillation

Also called knowledge distillation, model distillation

Training a smaller model to reproduce a larger model's outputs, aiming to keep much of the larger one's ability at a fraction of the cost to run.

See also

  • Parameters

    The numbers inside a model, its weights and biases, that are set during training and together hold what it learned.

  • Fine-tuning

    Further training of an already trained model on a smaller, focused set of examples so it gets better at a particular task or style.

  • Inference

    Running a trained model to produce an output, as opposed to training it.

  • Open weights

    A model whose trained parameters are published for anyone to download and run.

Source: Hinton, Vinyals and Dean, Distilling the Knowledge in a Neural Network (2015) (opens in a new tab)

Cite this term.

CC BY 4.0, attribution required.

This term

Orbyt Labs, Distillation (AI Glossary), Oct 18, 2026. https://www.orbytlabs.ai/orbyt-collective/glossary/distillation

The dataset (JSON)

Orbyt Labs Glossaries (Oct 19, 2026), https://www.orbytlabs.ai/glossary#dataset

Last updated .