LLM-as-a-judge
Evaluation method that uses a large language model to score the output of an AI system, including answers to open-ended questions.
In plain language
A language model grades the answer produced by an AI system.
In a sentence
- The team used LLM-as-a-judge to score the generated answers.
- We compared LLM-as-a-judge scores with the human reviewers' judgments.
Usage note
The score is a model's judgment of an output. Agreement with human reviewers is something to measure rather than assume.
LLM-as-a-judge
Guidance recommends a practice without making it a binding rule. Anthropic recommends closely calibrating model graders with human experts.
Source: Anthropic, agent evaluations 2026 (opens in a new tab)
See also
- Evaluation
Measurement of an AI system on specified tasks against stated criteria, using a repeatable set of inputs and a way to judge its outputs or actions.
- Chain-of-thought monitorability
Extent to which a monitor can use a reasoning model's chain of thought to predict a property of its behavior, such as misbehavior.
- Evaluation awareness
Ability of an AI model to recognize that it is in a test setting rather than being used in a real deployment.
- Constitutional AI
Method of training an AI model to be harmless in which human oversight consists of rules or principles, without human labels identifying harmful outputs.
Cite this term.
CC BY 4.0, attribution required.
Orbyt Labs, LLM-as-a-judge (AI Glossary), Nov 14, 2026. https://
Orbyt Labs Glossaries (Nov 14, 2026), https://
Last updated .