The first stage of training, in which a model learns general patterns from a large, broad dataset before it is adapted to particular tasks. For language models a common objective is predicting the next token, and later stages, such as fine-tuning and RLHF, shape it into a useful assistant.
See also
- Fine-tuning
Further training of an already trained model on a smaller, focused set of examples so it gets better at a particular task or style.
- Large language model
A model with a very large number of parameters, trained on very large amounts of text to predict the next token, which lets it write, summarize and answer questions.
- Knowledge cutoff
The date through which a model's knowledge is most extensive and reliable, which Anthropic calls its reliable knowledge cutoff.
- Compute
The processing power used to train and run AI models, often counted in hours on chips such as GPUs.
Cite this term.
CC BY 4.0, attribution required.
Orbyt Labs, Pretraining (AI Glossary), Oct 18, 2026. https://www.orbytlabs.ai/orbyt-collective/glossary/pretraining
Orbyt Labs Glossaries (Oct 19, 2026), https://www.orbytlabs.ai/glossary#dataset
Last updated .