AI Glossary

LLM-as-a-judge

noun phraseAI evaluationpreferred in this glossary

Evaluation method that uses a large language model to score the output of an AI system, including answers to open-ended questions.

Source: LangChain, model judges 2026 (opens in a new tab) (adapted)

Adapted: adds open-ended questions from the study by Zheng and colleagues. Source: Zheng et al. 2023 (opens in a new tab)

In plain language

A language model grades the answer produced by an AI system.

In a sentence

  • The team used LLM-as-a-judge to score the generated answers.
  • We compared LLM-as-a-judge scores with the human reviewers' judgments.

Examples written for this glossary

Usage note

The score is a model's judgment of an output. Agreement with human reviewers is something to measure rather than assume.

See also

  • Evaluation

    Measurement of an AI system on specified tasks against stated criteria, using a repeatable set of inputs and a way to judge its outputs or actions.

  • Chain-of-thought monitorability

    Extent to which a monitor can use a reasoning model's chain of thought to predict a property of its behavior, such as misbehavior.

  • Evaluation awareness

    Ability of an AI model to recognize that it is in a test setting rather than being used in a real deployment.

  • Constitutional AI

    Method of training an AI model to be harmless in which human oversight consists of rules or principles, without human labels identifying harmful outputs.

Cite this term.

CC BY 4.0, attribution required.

This term

Orbyt Labs, LLM-as-a-judge (AI Glossary), Nov 14, 2026. https://www.orbytlabs.ai/orbyt-collective/glossary/llm-as-a-judge

The dataset (JSON)

Orbyt Labs Glossaries (Nov 14, 2026), https://www.orbytlabs.ai/glossary#dataset

Last updated .