Inference

See also

• Compute

  The processing power used to train and run AI models, often counted in hours on chips such as GPUs.

• Test-time compute

  Computation a model spends while answering rather than during training, for example reasoning at length, sampling several answers or searching among them.

• GPU

  A chip first built to draw graphics, whose ability to do many calculations at once made it widely used for training and running AI models.

• Distillation

  Training a smaller model to reproduce a larger model's outputs, aiming to keep much of the larger one's ability at a fraction of the cost to run.