A failure that escaped becomes a numbered lesson. A lesson that bit becomes a guard, and the guard is proven loud and quiet. The practice behind the guards is harness engineering. This page is the other half: counting who the record says caught each failure. Mostly, it was not the guards.
An eval here is not a benchmark score. It asks whether the agents would make yesterday's mistake again, and whether anything would notice. Of the 26 incidents on record, seven were recorded as caught by the Machine's own checks, eleven by model sessions, three by the founder, and five were found in production. All but one of the 26 reached main or production first.
What is an agent eval at Orbyt Labs?
Two things. The first is a smoke eval of each seat's report against a baseline kept in the repository. It is structural: it checks the frontmatter, the headings, forbidden patterns and the word count, and it never reads what the prose means. An eval goes stale when its sources change or a newer completed run exists, not when the clock moves.
The second is the regression flywheel. A failure becomes a numbered lesson. A lesson that bit becomes a guard. The guard is shown to fire on the defect and to stay silent on correct input. That loop is most of this page.
Neither one is a benchmark. A benchmark asks how good a model is. This asks whether one running organization repeats its own mistakes.
What can this page not see?
The limits come first, because every count below depends on them.
- One organization, one repository, one founder. There is no control group.
- The trailers are written by the agents. A lesson that says a guard caught it is a claim; the harness-written evidence behind it, a run id or a hook log line, is not yet required on every row.
- The older counter groups model sessions and the founder together. This page reports them separately. A working session means a model session. The founder's catches that never reached a trailer are NOT OBSERVED here.
- For lessons 73 to 85, the split between model sessions and the founder was written in one later commit on , from trailers that had said human. One of them (lesson 76) moved from found in production to a model session.
- Time to guard is when a guard landed, not whether it held. Whether it held is NOT OBSERVED on this page.
- The nightly replay could score two of 241 run records on its latest night. Its verdict on the other 239 is NOT OBSERVED.
- This page evaluates an organization's process, not a model. Orbyt Labs does not run model evals, red-team models or measure alignment.
What is the evaluation pipeline?
Six stages, each a file in the repository, so someone else can run the same method on their own record.
- A failure that escaped ends its lesson with one trailer line: when it was introduced, when it was detected, what detected it, whether it escaped, when it was guarded, and which earlier lesson it repeats. The trailer is required from lesson 73 on.
- A checker parses every trailer and prints the table as JSON. It is the one reader of the convention; nothing re-parses the prose.
- A generator turns that table into the figures on this page. It writes nothing on a malformed trailer or a shallow clone, where git dates read wrong.
- A smoke eval checks each seat's report against its baseline.
- A commit can name a test in a Negative-Case trailer, and CI replays that test on the parent tree and on the commit, and reports, without blocking, whether it failed and then passed.
- A nightly replay scores a proposed rule change over the saved run records before anyone adopts it.
How does a failure become a guard?
By three rules written after seven errors in one day sorted into two classes. Every detector ships with a case it must not flag. Done means the artifact was read back, not that the file was edited. And the specific miss is mechanized, because a lesson is a hope and a guard is a fact.
The bar keeps the flywheel from turning into noise. A guard is added for a class that has actually bitten, with the receipt. Never for every hypothetical.
Most guards land on the day the incident was detected. The median time from detection to the guard is zero days, across all 26 trailers.
A guard can pass for the wrong reason. Lesson 63 records one that read the wall clock and stayed green at 1:00 AM Central with the fix deleted, because the gate it guarded then opened at 8:00 AM. "It passes" is not "it can fail."
How is a guard proven loud and quiet?
Loud means the defect is injected and the guard fires. Quiet means correct input goes in and the guard stays silent. A guard that has only ever been loud has not been tested. Four detectors misfired on , and every one of them had a green suite.
On the proof moved from the agent to the harness: the Negative-Case trailer. The founder ruled it report only for its first week, under decision D-0062.
It has been used once. One commit since , out of about 1,750 counted on , carries the trailer. The quiet proof still lives mostly in test files the agents write.
How many guards stay quiet across the whole suite is its own measurement, in Every Guard Must Stay Quiet.
Who does the record say catches the failures?
The trailer table holds 26 incidents, lessons 73 to 98, detected between and , except lesson 91, written up later about an incident detected . Seven are recorded as caught by the Machine's own checks: four by a test, two by a guard and one by an outside model's check. Eleven are recorded as caught by model sessions and three by the founder. Five were found in production.
All but one of the 26 reached main or production first. The median incident sat for eleven days between the commit that introduced it and the day it was detected, over the 25 with a known start.
The guards are mostly built after the fact. That is the honest shape of a regression flywheel: it is built for the second time. Failures that raise no error at all are the subject of Confidently Wrong.
The standing record of who catches what, across every kind of issue and not only escaped incidents, is the AI Oversight Ledger: 97 catches as recorded through .
Who caught each incident, by month
September 2026 holds 20 of the 25 incidents drawn. In it, 11 are recorded as caught by a model session or the founder and 5 by the Machine's own checks.
Recorded as caught by the Machine's own checks: a test, a guard or an outside model's checkRecorded as caught by a model session or the founderFound in production; the trailer names no catcher
Move over a month, tap it, or focus the chart and use the arrow keys.
- Every lesson from 73 on must carry a trailer. An incident written up in an earlier lesson, or never written up, may be missing from the table.
- Most of the drawn incidents fall in September, so the other months rest on 5 incidents between them.
Not shown. One incident detected before the first month drawn and written up later (lesson 91, detected ). It has no column, because the table does not say what else those months held. The table below and the downloads list it.
Built from incident-figures.generated.json, which scripts/research/method-figures/agent-evals.ts writes from the trailer table that node scripts/check-incident-trailers.cjs --json prints from docs/lessons-learned.md.
Newest incident detected
The numbers as a table
| Month | Machine's checks | Model session or founder | Found in production | All |
|---|---|---|---|---|
| June 2026, not drawn | 0 | 1 | 0 | 1 |
| September 2026 | 5 | 11 | 4 | 20 |
| October 2026 | 2 | 2 | 0 | 4 |
| November 2026 | 0 | 0 | 1 | 1 |
Download the data: agent-evals-who-caught.csv, agent-evals-who-caught.json
An earlier draft printed a single September window: 13 incidents detected September 12 to 22, 2026, a guard catching 2, and 7 under the old label human, which grouped model sessions with the founder. It is kept only as a dated fact. The table above supersedes it.
Did the flywheel stop failures from coming back?
Not on the record, yet. Nine of the 26 trailers name an earlier lesson as the same class of failure.
For 3 of them the record shows the earlier guard. Two landed after the new defect was introduced. One, lesson 94's, was already in place, and the class came back anyway (lesson 95). For the other 6 the record has no guard date.
Whether a failure returns less often after its guard, against the chances it had to return, is a preregistered question with its own stop rule: Did the Machine Learn. This page does not answer it.
How do we red-team our own agents' work?
By assigning reviewers whose job is to prove the work wrong. Four practices, each in the repository:
- Refuters whose default stance is refuted. The showcase review sends finders across a surface, then three refuters per surface; a finding survives only if at most one of the three kills it.
- Cold reviews in a fresh context that execute their negative cases, restoring the old code to watch the check fail, before a change lands.
- A red-team agent for long-running arcs, forked into its own context by the redteam command.
- Every guard proven loud and quiet, so a check that cannot fail is caught as decoration.
The committed review series holds 26 cold reviews of five designs. All 26 found blocking issues. Every reviewer was a Claude sub-agent. The record does not say who wrote the designs.
Dissent is assigned at the board level too. The Tenth Man argues the board is wrong whenever it agrees, and its rule and its record are on Synthetic Personas. When the writer and the reviewer are different models, the method is cross-model review.
This is adversarial review of the agents' work: their code, their reports, their claims. It is not red-teaming of models, and no finding here is about a model's safety.
When is a model the judge?
When the question is judgment, and never when it is a count. A model grades taste, argument and polish: the showcase review's letter grade, a cold review's verdict, a panel's grade of a page. Each is kept as a judgment, never as a count.
A model never computes a published figure. Every number on this page is read from the committed record by a script or a named command, and the daily pulse is a script with no model in it, so none of its figures comes from a model.
The judges are checked by other judges, not by ground truth. A refuter that wrongly kills a true finding leaves no trace in these counts.
The showcase review is report only unless it is run strict, so a model judge's grade blocks nothing by default. How often the judges agree with the founder is NOT OBSERVED.
What is the nightly policy replay?
A way to test a rule change before anyone adopts it. The replay takes a proposed policy, one of a skip, a turn cap, a review budget or a loop stop, and scores it over the saved run records as if it had been in force.
The daily pulse runs one standing replay each night: a cap of 100 turns on every seat. The replay proposes. It never adopts. When it cannot compare, it says not observed instead of reporting a clean result.
That last rule is doing most of the work. There are 49 nightly receipts, from to by their file stamps in UTC. On the latest one the replay read 241 run records and could score two of them, because 152 do not record their turns in the cap's unit and 87 ran under a lower cap, which the replay will not raise. Twelve nights gave no verdict and three nights have no receipt, so those fifteen were not observed.
The replay scored 9 to 12 runs a night from through , gave no verdict on every night with a receipt from through , and has scored 1 or 2 on every night with a receipt since . , and have no receipt. The scorer began requiring each run to record its turn unit on (commit 840c7ce26), and the first night after it with a verdict was .
Run records the nightly replay read and could score
On its newest night the replay could score 0.8% of the run records it read, against 7.3% on its first.
Run records read that nightRecords it could scoreNo receipt that nightStrip below: no verdict that night
Move over a night, tap it, or focus the chart and use the arrow keys.
- A record the replay could not score has no verdict. The receipts say so themselves: excluded rows are not zero.
- Dates are the receipts' file stamps in UTC, so a night's date can be the day after the evening it ran in Central time.
Not shown. Why each other record could not be scored. Each receipt gives its reasons in its not_scorable and coverage blocks.
Built from incident-figures.generated.json, which scripts/research/method-figures/agent-evals.ts writes from the nightly receipts in docs/org/replays/.
Newest receipt
The numbers as a table
| Night | Records read | Records scored |
|---|---|---|
| 123 | 9 | |
| 125 | 10 | |
| 131 | 10 | |
| 135 | 10 | |
| 139 | 11 | |
| 141 | 11 | |
| 143 | 11 | |
| 146 | 12 | |
| 149 | 12 | |
| 151 | No verdict | |
| 153 | No verdict | |
| 155 | No verdict | |
| 158 | No verdict | |
| 158 | No verdict | |
| 163 | No verdict | |
| 165 | No verdict | |
| 167 | No verdict | |
| 169 | No verdict | |
| No receipt | Not observed | |
| 170 | No verdict | |
| 173 | No verdict | |
| 175 | No verdict | |
| 178 | 1 | |
| 180 | 1 | |
| 183 | 1 | |
| 185 | 2 | |
| 188 | 2 | |
| 190 | 2 | |
| 192 | 2 | |
| 192 | 2 | |
| 195 | 2 | |
| 196 | 2 | |
| 199 | 2 | |
| 202 | 2 | |
| 206 | 2 | |
| 208 | 2 | |
| 208 | 2 | |
| No receipt | Not observed | |
| 212 | 2 | |
| 215 | 2 | |
| 217 | 2 | |
| 219 | 2 | |
| No receipt | Not observed | |
| 224 | 2 | |
| 225 | 2 | |
| 228 | 2 | |
| 230 | 2 | |
| 232 | 2 | |
| 234 | 2 | |
| 237 | 2 | |
| 238 | 2 | |
| 241 | 2 |
Download the data: agent-evals-replay-nights.csv, agent-evals-replay-nights.json
The rule that bounds the replay is decision D-0062: the Machine improves itself inside a fixed constitution, and the loop proposes rule changes and never adopts them. A replay that can score almost nothing proposes almost nothing, which is the right output for a record that cannot answer.
Agent eval questions? Answers.
Related reading
- Harness engineering
How agent mistakes become standing instructions, tests, checks and gates.
- AI oversight ledger
The standing record of who caught each issue and who fixed it.
- Synthetic personas
The lens panel: the synthetic reviewers the agents convene, and what the lenses caught.
- Did the Machine Learn
The preregistered question of whether a failure returns less often after its guard.
- All Orbyt Labs methods
Every method page, each with its dated public record.
Cite this page
Cite this page for the practice and follow its sources for the underlying records.
Bartak, J. (2026). Agent Evals. Orbyt Labs Research. https://