nounAI evaluationpreferred in this glossary

Evaluation metric for the probability that an AI system produces at least one correct solution in k attempts at a task, rather than requiring every attempt to succeed.

Source: Code evaluation paper (opens in a new tab) (adapted)

Adapted: uses the probability framing stated in the cited benchmark's explanation of repeated task trials. Source: Tau-bench paper (opens in a new tab)

In plain language

Count whether any attempt succeeds within the allowed tries.

In a sentence

  • The report gave pass@k alongside the number of attempts allowed per task.
  • A high pass@k did not mean the agent succeeded on every attempt.

Examples written for this glossary

Usage note

Success at least once and success on every trial answer different evaluation questions. State which event the reported score measures.

  • pass^k

    Research

    A research publication reports a question, method and findings. The benchmark defines pass^k as the chance that all k independent, identically distributed task trials succeed, averaged across tasks.

    Source: Tau-bench paper (opens in a new tab)

See also

  • Benchmark

    Repeatable collection of tasks and scoring measures used to compare the performance of AI systems under stated conditions and on a stated range of work.

Cite this term.

CC BY 4.0, attribution required.

This term

Orbyt Labs, pass@k (AI Glossary), Nov 14, 2026. https://www.orbytlabs.ai/orbyt-collective/glossary/pass-at-k

The dataset (JSON)

Orbyt Labs Glossaries (Nov 14, 2026), https://www.orbytlabs.ai/glossary#dataset

Last updated .