A public, dated record from Orbyt Labs: failures we observed in a codebase that AI agents operate, the checks that followed, what they do and where they stop.
What is harness engineering?
In harness engineering, when an agent makes a mistake, the correction goes into standing instructions, a test, a check or a gate. The next run no longer depends on someone remembering the last conversation. Before you trust a check, watch it catch the defect and pass correct work.
Prompt engineering changes the request. Model training changes the model. Harness engineering changes the working environment and the checks that judge the output.
Which harness this page means
The field uses the word more widely. Anthropic's guide to agent evals calls the agent harness the system that enables a model to act as an agent, and its writing on long-running agents treats how work survives from one session to the next as harness work. This page covers a narrower part: the checks and corrections that follow a failure. It does not cover context management, session continuity or the agent loop itself. Sources: Anthropic: Demystifying evals for AI agents (opens in a new tab) and Anthropic: Effective harnesses for long-running agents (opens in a new tab).
These checks do not measure agent reliability
A check asks whether a known defect is present in one piece of work. It is built to give the same answer on the same input; when one has not, the record files that as a failure of the check. The incidents on this page demonstrate particular regression checks, without estimating task success rates or the effect of changed instructions. Most agent seats' reports are also checked by code, without a model, for their shape: required fields and headings, banned patterns and length. That checker never judges whether a report is right and does not stop the work from landing, and it is tested on deliberately bad reports that it must fail. It is one format check of one report from one run. This record has no repeated-trial measure of whether a changed instruction changed how often an agent errs.
The preregistered study plans two measures: a repeated-trial comparison of a raw incident record against its condensed lesson, on 3 task families with 3 paired repetitions each, up to 18 attempts in all, and a test of each included check, once frozen, on a variant of its failure it has never seen. Attempts run: NOT OBSERVED, because the folder where runs would be recorded does not exist yet. That is not a count of zero, and the study has reported a result for neither. Read the preregistration.
What is not new
None of these techniques is new: regression tests, CI gates, mutation testing, negative controls, provenance tracking, least privilege, independent monitoring, standing instructions, tools that correct mistakes, separate measured checks and subjective review, human authorization for consequential gates, and review by multiple models. What this account contributes is the incidents, corrections and limits from running them around agents.
What this record keeps
A check needs evidence too. So it keeps:
- Failures of verification, recorded alongside failures of the work.
- A refusal that succeeded while its explanation failed.
- A check the agent it judged could rewrite.
- A protective check that disabled the monitoring and its repair.
- A dated chain from incident to named countermeasure.
- A preregistered test, not yet reported, of whether stored lessons reduce recurrence.
How does Orbyt Labs practice harness engineering?
Orbyt Labs practices harness engineering in four steps: record the failure, write the numbered lesson, turn the lesson into a check, and decide who may change the checks.
Record the failure.
Keep what went wrong and the evidence that exposed it. The public failure corpus gives each recorded incident an identifier, a category and a countermeasure where one exists.
Read the failure corpusWrite the numbered lesson.
The full lesson is recorded internally. Its public version carries the same number in the failure corpus. The log-status incident, for example, became a lesson about checking that two outputs agree.
Read the log-status accountTurn the lesson into a check.
Write a test or an audit dimension, a check for a whole category of failure. Guards are the scripts that run these checks. Inject the defect and watch it fail: the loud case. Give it correct input and watch it stay quiet: the quiet case. A check that has only passed has not proved it can catch anything.
Read the audit dimension guideDecide who may change the checks.
Agents may add audit dimensions that only report a problem. Promoting one to a gate that blocks commits takes the owner's sign-off. Tests are different: a test written in an attended session joins the suite and can stop a push from its first red run. An agent working unattended cannot publish a change to any test, new or existing, or to a guard script or judging module on the landing gate's list: the gate refuses it, pushes nothing, opens no pull request and keeps the commits for a person. A change to any other file outside the gate's closed allow list is held as a pull request for a person to review. The agent seats' writable lanes include no tests or guards at all. Finding a defect and deciding who may stop a change are separate decisions.
Read the authority rules
The record in numbers
A failure can be a verification gap and also have a named countermeasure, so these counts overlap.
- Documented failures
- 97Detected and recorded by one organization.
Counted
- Classified verification gaps
- 53Of 97 documented failures. Operator classifications.
Counted
- Named countermeasures
- 73Of 97 documented failures. The count confirms a named file exists. The generator does not run it.
Counted
Check inventory
- Audit dimensions
- 100
Assembled on
- CommonJS check scripts
- 119
Assembled on
- Recorded test inventory
- 55,942Mixed counting methods and observation dates, not an executed combined suite. Component observation dates are unknown where the artifact does not record them. The history below counts the main repository's suite alone.
Assembled on
These are recorded inventory counts. They do not measure useful coverage or prevented failures. Guards are scripts; audit dimensions are categories.
Test inventory sources and observation dates
- Orbyt Website: 7,550. Source-pattern counts of unit and browser tests. Observation date: not recorded in the artifact.
- Orbyt Collective: 4,595. Source-pattern count of framework tests plus console tests. Console counts may be carried forward; this artifact does not identify whether they were. Observation date: not recorded in the artifact.
- Jobs: 38,440. Cached repository suite count, or a source-pattern fallback, minus other unit partitions, plus browser source-pattern counts. The artifact does not identify which input was used. Observation date: not recorded in the artifact.
- Intelligence: 1,435. Source-pattern counts of unit and browser tests. Observation date: not recorded in the artifact.
- Orbyt One: 304. Source-pattern counts of unit and browser tests. Observation date: not recorded in the artifact.
- Orbyt Mobile Apps: 22. Source-pattern count of Swift test methods. Observation date: not recorded in the artifact.
- Orbyt Arcade: 49. Source-pattern counts of unit and browser tests. Observation date: not recorded in the artifact.
- Orbyt Books: 29. Source-pattern count of selected unit tests. Observation date: not recorded in the artifact.
- Orbyt Agent Fleet: 3,518. Separately dated agent repository receipt. Observation date: .
A compact history
Committed observations of the main repository's test suite and audit roster, oldest first.
- : 63 tests in the main repository's suite; 0 audit dimensions. Timeline record
68739423d - : Pre-commit script first committed. History record
4b02942af - : Log-truth check. History record
8de974a1b - : 46,651 tests in the main repository's suite; 100 audit dimensions. Timeline record
b7b44b750
Each recorded failure, by class and date
One dot for each failure in the public corpus, in its class's row and the week of the first date mentioned on its numbered lesson line. A filled dot names a countermeasure. A hollow dot names none. Hover, drag or tap the chart, or focus it and use the arrow keys, to read one failure.
Ordered by first date mentioned, through failure 97, first date mentioned
- Countermeasure named
- None recorded
Failure 97. First date mentioned in the lesson . Verification gap.
Every public page shipped its content inside a hidden streaming block that a script swaps into place after load, because a route-wide loading boundary that rendered nothing wrapped them all, and React moves a large finished boundary out of line even on a prerendered page; browsers showed the pages and another company's AI read the blog as empty, while a link count over the raw HTML called it healthy
Countermeasure: __tests__/structure/site-content-in-html.test.ts
73 of 97 recorded failures name a countermeasure, and 24 name none. 86 have a first date mentioned from to ; 11 carry no date in their record and sit in their own column. The latest first date mentioned among failures with no named countermeasure is .
What this chart does not show. Which check caught a failure: the corpus records the fix, not the catcher. When a failure happened or was documented: the date is only the first date mentioned on its numbered lesson line, which may describe occurrence, repair or supersession. Whether a countermeasure works: a filled dot means a file is named, and the corpus generator confirms the file exists without running it.
Target, not a measurement. The goal stated under Where this is going is that no error goes unseen or unhandled. This page sets no numeric target, so the chart draws none. Read the target.
Counted
The failures behind the chart
| Failure | Title | First date mentioned in the lesson | Class | Countermeasure |
|---|---|---|---|---|
| 1 | A mobile browser can suspend a tab without releasing its Web Locks, deadlocking auth refresh | Undated | External behavior | app/layout.tsx |
| 2 | Realtime sync can echo a write back into itself unless transient UI state stays out of the database | Undated | External behavior | none recorded |
| 3 | High-frequency writes to a replicated column can destabilize the replication layer itself | Undated | External behavior | none recorded |
| 4 | A cleared cache with a surviving session is a distinct state; UI must wait for hydration to complete | Undated | Operator process | none recorded |
| 5 | A managed provider's default email path cannot be assumed to deliver; route through one you can observe | Undated | External behavior | none recorded |
| 6 | Unsafe HTML injection requires an explicit, same-line trust annotation to pass review | Undated | Operator process | none recorded |
| 7 | A module whose internal name diverges from its export breaks tooling that assumes they match | Undated | Operator process | none recorded |
| 8 | After a rename, the old name is a liability: redirect it once, then never reference it again | Undated | Operator process | none recorded |
| 9 | A date test that mocks the clock in a different timezone convention than the code parses fails only west of UTC | Undated | Verification gap | none recorded |
| 10 | A client library that auto-deserializes on read breaks every consumer that stored a string, and only on cache hit | Undated | External behavior | app/api/_lib/cache.ts |
| 63 | A guard that reads the wall clock is vacuous at the wrong hour, so deleting the fix it protects can leave it green | Undated | Verification gap | __tests__/site/scheduled-content-containment.test.ts |
| 91 | A connector fetched a binary archive through a text reader, so the UTF-8 decode destroyed about 74,000 bytes and the parser blamed the file; it now fetches bytes behind the same guards, and a test holds the byte path | Verification gap | __tests__/intelligence/ingestion/http-bytes-json-guards.test.ts | |
| 11 | A programmatic page matrix needs a bounded kept-set imported by both the builder and the sitemap | Operator process | __tests__/site/salary-data.test.ts | |
| 12 | An internal spec date is not a protocol version; clients hard-reject unknown values | Operator process | __tests__/mcp/protocol-negotiation.test.ts | |
| 13 | A redirect parameter is a contract between writer and reader; a silent fallback hides the mismatch | Operator process | none recorded | |
| 14 | Hiding overflow on one axis creates a scroll container on the other and can trap wheel events | External behavior | none recorded | |
| 15 | A client-reachable module that statically imports a heavy data source ships it to every page that renders it | Operator process | none recorded | |
| 16 | A test that derives its expected value from the code under test can never fail | Verification gap | scripts/check-log-status-truth.cjs | |
| 17 | A pre-authentication code path needs its own abuse ceiling | Operator process | app/api/v1/intelligence/_lib/route-wrapper.ts | |
| 18 | A table listing's row counts are planner estimates; report a real count or report nothing | Verification gap | none recorded | |
| 19 | Email clients default to content-box, so full width plus padding overflows a phone | External behavior | none recorded | |
| 20 | A CI gate that only runs on the day it matters has never actually been tested; force the condition once | Verification gap | none recorded | |
| 21 | A convention that has never been executed is not a working convention | Verification gap | none recorded | |
| 22 | A build filter that decides inclusion is a detector, and must be proven on the shortest legitimate case | Verification gap | none recorded | |
| 23 | Invalid CSS is discarded, not flagged; a broken value silently becomes the property's initial value | External behavior | none recorded | |
| 24 | Read a provider response by scanning for the block you want, never by indexing position zero | External behavior | scripts/check-ai-response-shape.cjs | |
| 25 | A model's output budget caps reasoning plus text; leave neither half implicit | External behavior | app/api/_lib/ai-models.ts | |
| 26 | A tool that runs quarterly has no gate between quarters, and its hand-copied config drifts silently | Verification gap | __tests__/config/stryker-config-sync.test.ts | |
| 27 | A cron reporting success is not healthy; watch duration against its ceiling and reap rows whose process never returned | Verification gap | none recorded | |
| 28 | A timeout reports as cancelled, not failed; alarms keyed on failure alone are structurally blind to it | Verification gap | scripts/check-ci-headroom.cjs | |
| 29 | A metric measured and then discarded is indistinguishable from a metric never measured | Verification gap | none recorded | |
| 30 | A synchronous spawn against an in-process fake server deadlocks, and the hang reads as the child failing | Verification gap | none recorded | |
| 31 | A step that discards stderr turns a missing permission into a silently wrong branch | Verification gap | __tests__/scripts/daily-pulse.test.ts | |
| 32 | A relabeled invariant must sweep every detector that encodes it, in the same commit; it recurred in September 2026, when a date constant moved to a new file, the integrity check kept reading the old one and reported it missing for eight days, and the Chief of Staff carried the false alarm into the briefing | Operator process | scripts/check-data-integrity.cjs | |
| 33 | Checks sharing one rate-limit window flake each other; absorb exactly one window or serialize | External behavior | scripts/check-smoke.cjs | |
| 34 | A fix committed but not pushed does not exist, and a guard grading the working tree will certify it as live | Verification gap | scripts/check-ci-headroom.cjs | |
| 35 | An instrument's failure paths must render as not observed, never as clean | Verification gap | __tests__/scripts/daily-pulse.test.ts | |
| 36 | A lesson mechanized in the canonical pipeline does not exist for the surfaces that bypass it | Operator process | __tests__/scripts/org-guard-parity.test.ts | |
| 37 | A selection rule optimizes its proxy; verify what exists against measured demand, not against the code | Operator process | scripts/check-salary-demand-coverage.ts | |
| 38 | A sync rule that tests equality can never be satisfied by the thing it exists to allow; record provenance instead | Operator process | __tests__/scripts/sync-i18n-provenance.test.ts | |
| 39 | Never validate a filter against data that filter produced | Verification gap | none recorded | |
| 40 | A gate whose correctness rests on the thing it is gating is not a gate | Verification gap | packages/collective/actions/org-agent/action.yml | |
| 41 | A job cancelled with zero steps was never handed to a runner; an alarm inside the job cannot see the job never starting | Verification gap | scripts/check-ci-headroom.cjs | |
| 42 | Read a monitor's freshness the way the monitor is actually fed | Verification gap | __tests__/scripts/daily-pulse.test.ts | |
| 43 | A stale document reads as instructions; when an invariant reverses, sweep the documents that assert it | Operator process | scripts/check-root-clutter.cjs | |
| 44 | A sandbox that cannot start reports success, and a fence can apply halfway | Verification gap | .github/workflows/fence-canary.yml | |
| 45 | A confinement lane is what an agent is instructed to write, not what gets committed | Operator process | __tests__/scripts/org-fence.test.ts | |
| 46 | A guard's exit code is not its verdict; report-only and strict-gated checks exit zero while reporting | Operator process | none recorded | |
| 47 | A gate that reads a tool's printed summary is reading a rendering, and colour in CI can leave it with no way to fail | Verification gap | __tests__/scripts/test-gate.test.ts | |
| 48 | Permissive access policies combine with OR, so one globally true policy defeats every narrow policy beside it, and two individually correct guards can still leave the class unowned | Verification gap | scripts/check-policy-breadth.cjs | |
| 49 | A generated file that measures the repository containing it can never be correct at the moment it is committed, and a drift guard that performs no regeneration of its own reports PASS on a precondition it never established | Verification gap | scripts/check-generated-drift.cjs | |
| 50 | A shallow repository clone makes the version-control history answer confidently and wrongly, so a generator that trusts it publishes a false freshness claim with no error anywhere | External behavior | scripts/update-skunkworks-dates.cjs | |
| 51 | A privileged key does not bypass a control the upstream service enforces, and charging that service's refusal to a user's abuse budget locks out the innocent | External behavior | __tests__/api/mobile-login.test.ts | |
| 52 | An upstream service reports a downstream failure in its own vocabulary, so the error names a cause that is not the real one, and four consecutive fixes were aimed at the wrong side of the wire | Operator process | scripts/check-auth-hook-health.cjs | |
| 53 | Internal documentation is not a wire contract: a set of event names transcribed from reviewed internal docs matched nothing the sender actually emits, and a single observed value disagreeing with a document invalidates the whole document rather than the one row it lands on | Operator process | __tests__/api/auth-send-email-hook.test.ts | |
| 54 | A monitoring query that samples without filtering reports the absence of the needle as the absence of the problem, and sorting newest-first does not fix it; a reported count equal to the page size is never a measurement | Verification gap | __tests__/scripts/check-auth-hook-health.test.ts | |
| 55 | A dependency's implicit fetch option is a route-classification decision, and it can silently repeal a page's own segment config, turning a declared 404 into a streamed 200 soft-404 that only the deployed build exhibits | External behavior | scripts/check-blog-route-static.cjs | |
| 56 | An allowlist entry is a claim about production that decays silently, and a guard checking whether a value is declared cannot go loud about the value that actually costs money | Verification gap | __tests__/site/dynamic-params-guard.test.ts | |
| 57 | A verification that normalizes its input passes on a value the real consumer will reject, so an instrument must read its subject exactly the way the consumer reads it | Verification gap | none recorded | |
| 58 | A cost measurement is valid only for the resource it measured on the machine that binds it, and a local build stops one whole stage before the one that fails | Verification gap | __tests__/site/dynamic-params-guard.test.ts | |
| 59 | A field that may contain markup is safe only where it is rendered as markup, and the sink no human ever looks at is the one that keeps shipping the defect | Verification gap | __tests__/site/faq-rendered-markdown.test.tsx | |
| 60 | A proximity window measured in lines stops measuring anything once a generator writes the file, and a framing verb inside a citation is not a claim of affiliation | Verification gap | __tests__/scripts/check-advisor-name-leak.test.ts | |
| 61 | A test that spawns an entrypoint aware of its build environment inherits the live reporting channel, so a suite can overwrite the run's own verdict and the last write wins | Verification gap | __tests__/scripts/runner-env-isolation.test.ts | |
| 62 | A hand edit to a generated file is one run from silent reversion, and a suite that grades the file instead of a regeneration will never see it | Verification gap | __tests__/scripts/untimed-heading.test.ts | |
| 65 | A shell capture of a non-string value from a runtime that colorizes its output returns ANSI escape bytes, and the environment that does it is the automation harness rather than a human terminal | External behavior | none recorded | |
| 66 | Calendar month arithmetic overflows into the following month, and a test that computes its expectation the same way agrees with the defect instead of catching it | Verification gap | __tests__/api/intelligence-pause.test.ts | |
| 64 | A corpus gated on the wrong date field publishes a placeholder before the enriched version can exist, and an exemption list written the day a feature lands is the tell | Verification gap | __tests__/site/scheduled-content-containment.test.ts | |
| 67 | A dependency that is never declared but resolved on demand by the package runner is invisible on any machine that has already cached it, and a lockfile matching its name is not evidence it is installed | External behavior | __tests__/mcp/cli-e2e.integration.test.ts | |
| 68 | A provider can retire a request parameter, so a model list is not a capability list; only a live call per model proves the request shape still works | External behavior | __tests__/api/openai-budget.test.ts | |
| 69 | A step's shell flags are part of the code, and a value written to the runner's environment file is for the next step, never the one that wrote it; a script's own tests prove the script, so the shell around it has to be executed too | Verification gap | __tests__/scripts/org-agent-shell-contract.test.ts | |
| 70 | A step's environment is not the whole job's environment, so converting some call sites to a resolved path and not others leaves a contract with one side missing; a syntax check reads grammar and cannot see a name | Operator process | __tests__/scripts/org-agent-shell-vars.test.ts | |
| 71 | A design amendment that is not swept into the contract it amends leaves a path that cannot be walked; a sweep by reading finds the sentences that mention the premise, a sweep by execution finds the rules that depend on it | Operator process | __tests__/scripts/collective-command-contracts.test.ts | |
| 72 | A committed artifact derived from a publish-gated set changes bytes with no commit, so a commit-time heal cannot see the clock move before the push, and a hand-copied hook block is a list nothing can read: one registry, regenerated at commit and re-derived at push | Verification gap | __tests__/scripts/unconditional-artifacts.test.ts | |
| 73 | A field performance metric measures whoever ran the script: a scraper fleet on residential proxies filed its own proxy latency under the site's page names, while two real layout shifts (a form absent from static HTML behind a Suspense fallback, a font stack that skipped its metric-matched fallback face) hid in the same week, invisible to the lab | Verification gap | __tests__/components/site/InterviewPrepTool.test.tsx | |
| 74 | A third-party client script is a contract that can change without a deploy: a bot-check widget was rendered with an option the provider later began rejecting, so every form silently posted no token and sign-in failed for everyone, while the end-to-end test asserted only that the token field existed, which a null value satisfies | Verification gap | __tests__/auth/use-turnstile-config.test.ts | |
| 75 | An alarm with a catch-all arm files the case it was built for as unexplained, and a verdict with no reason is a log entry, not a record: an agent closed a run reporting every guard green, the harness's own gauntlet refused the tree, and the refusal reached no reader, because the outcome row carried no reason, the failing guard's name lived only in the job log, and the failure alarm had no arm for the most common way a run ends red | Verification gap | __tests__/scripts/seat-workflow-generation.test.ts | |
| 76 | A claim guard that reads source cannot see a contradiction between two published pages, a hand-typed number beside a generated one, or a gate applied to one surface and not its sibling; twelve contradictions stood with every guard green because each guard read one file, and only the page the visitor actually receives can show them | Verification gap | scripts/check-public-clarity.cjs | |
| 77 | A guard that audits the machine watched the repository's build cache and not the directory the test suite fills; 213,400 sandboxes made by mkdtemp and never removed held 40 GB while the guard named a 0.8 GB cache, so the temp directory's count joined the guard and every sandbox a test file makes is now removed when the file ends | Verification gap | __tests__/helpers/sandbox-reaper.ts | |
| 78 | A root-anchored ignore pattern left every nested node_modules trackable, and 115 installed files rode in the repository for a month with every guard green because on main the versions agreed; the first dependency bump let npm dedupe them away and the drift guard reported 115 tracked deletions as generator drift, so the index is now read directly | Operator process | __tests__/structure/no-tracked-node-modules.test.ts | |
| 79 | A main branch red at the base for a day billed every seat run inside the window as a guard refusal of its own work, kept their bookkeeping rows off main because the row's commit ran the same red suite, and let the briefing report three unrelated instruments dark when there was one cause; two of the instrument's own reads were also wrong, a by-design null rendered as unreadable and an unbounded run listing stamped fresh | Verification gap | __tests__/scripts/gh-snapshot.test.ts | |
| 80 | A prompt fix told the agent to iterate with a command that opens a socket, and the fence the agent runs inside refuses sockets, so the first walk of the fix could not execute the one command it consisted of; the agent rebuilt the tool by hand, polled a minutes-long suite five times, met two tests that write their scratch directory inside the repository, and hit its turn cap with a publishable draft in the tree | Operator process | __tests__/scripts/seat-prompt-fence-safe.test.ts | |
| 81 | A hook exports the repository it is operating on into the test suite it runs, and only from a linked worktree, so every sandboxed git command in the suite targeted the real repository while a reproduction on a plain clone was clean; the class had been fixed three times, once per file, and the fix now lives at the one setup joint every test file passes through | Verification gap | __tests__/helpers/git-env-isolation.ts | |
| 82 | A manuscript switch repointed the two constants that read the book log and none of the thirty-five surfaces that name it, so for sixteen days every seat run wrote to the closed book by every rule it could read and the published ledger counted the entries as that book's; the fix is one declaration and a guard that holds every copy to it | Operator process | __tests__/scripts/active-book-log-parity.test.ts | |
| 83 | A guard that restored whatever changed during its run in the real repository reverted a developer's concurrent edits with nothing printed, and a module tested by one runner measured as untested by the runner that grades coverage; the fix restores only what was snapshotted, names the rest, and makes the local gate measure what CI measures | Verification gap | __tests__/scripts/check-generated-drift.test.ts | |
| 84 | A rebase resolver's rule that no generator it runs may need an install lived in a comment, verified once by hand; a require added two files below a generator broke it, the conflict path ran the generator for the first time in a seat with no node_modules, and a complete briefing was discarded over two derived JSON files while the stats row's own rebase had no resolver at all; the fix reads the one dependency two ways, walks every generator's graph in a test, and gives the stats row the same resolver | Verification gap | __tests__/scripts/generator-runtime-purity.test.ts | |
| 85 | Three full-viewport fixed layers, one mounted invisible at opacity zero with eight overflow-scroll panels inside it and one an animating canvas, were bitmaps iOS re-rasterized under every pinch until WebKit's memory limit killed the page on every marketing page for six months; no desktop tool or simulator could see it, and the canvas was twice recorded as exonerated because every probe that survived with it had loaded with its JavaScript dead, so the subject under test had never mounted | External behavior | __tests__/site/nav-drawer-visibility.test.ts | |
| 86 | A check that reads a pipeline under pipefail, with an early exiting reader such as grep -q behind a writer, takes the writer's broken pipe signal as its verdict: a landing check reported a landed commit as missing only under load, and a journal guard passed the dash it had just found; every site now reads to the end, and a list larger than any pipe buffer makes the race certain in the test, so it stops being a flake | Verification gap | __tests__/scripts/journal-dash-guard-sigpipe.test.ts | |
| 87 | bash reads a workflow step's script from disk as it runs, so every check placed after an AI agent in the same step was a line the agent could rewrite before bash reached it; three holds built that week were bypassable until the step became one brace group ending in exit, which bash parses whole before running any of it | External behavior | __tests__/scripts/step-script-rewrite.test.ts | |
| 88 | A workflow step's script that names even one input through the runner's embed syntax is read by the platform as a single expression capped at 21000 characters; the agent step's script grew past it and every seat run died before its agent started, while the suite stayed green because its tests substituted those embeds out and executed a script the runner never sees; the inputs now travel through the step's environment, and a test grades every workflow file against the cap | External behavior | __tests__/scripts/expression-length.test.ts | |
| 89 | A job that installs the git hooks and then commits runs the whole test suite inside its own time ceiling, and the ceilings were set when the suite was smaller; a model refresh was killed inside its commit and the drift it found reached nobody, and three more jobs held the same ceiling on a committing path that had never run; each now has room, and a test holds every such job's ceiling to the suite it carries | Verification gap | __tests__/scripts/hooked-commit-ceiling.test.ts | |
| 90 | A test drove the push-time drift check with a planted generator and no sandbox root, so the plant sat in the real repository for the length of the run; a parallel test that had pinned the generators but not the root graded it and failed under load, and in a forced race one run's restore wrote the plant back after the other had removed it; both now grade repositories they own, and a structure test holds every test that reaches the check's writing modes to a pinned root or a case that says, with a reason, that it runs against the real tree | Verification gap | __tests__/structure/live-tree-push-check.test.ts | |
| 92 | A scheduled route's deployed function grew to 1.15 GB against a 250 MB limit and the deploy was refused, because it imported a script whose command-line entry reads the process's own path and Next's tracer answers that by packaging the whole repository; the logic now lives in a pure module the route imports, and a test walks every API route's import graph and refuses a filesystem call fed from the launch arguments or the module's own URL | Verification gap | __tests__/scripts/function-import-graph.test.ts | |
| 93 | A leak guard that refused by discarding the whole daily report left pulse and improvement-record artifacts 51 hours stale after two failed daily runs over one long number in a third party’s run title, and the alarm, the repair trigger and the watcher were all inside or beside the thing that failed; free text is now redacted at one boundary, the failed generator’s real error is printed, a watchdog outside the job reads the report’s age from the remote, and a failed run wakes the repair worker | Verification gap | __tests__/scripts/public-text.test.ts | |
| 94 | A test suite that was green on the developer’s machine killed the CI runner and then failed two dozen cases there, for three platform reasons at once: a process id read from a file that was never written became 0 and signalled the test runner’s own process group, a BSD-first command probe succeeded silently on Linux where the same flag means something else, and the runner’s preinstalled interpreter was world-writable, which the confinement layer correctly refused; signals now require a pid above 1, platform forms are chosen by the operating system, and the tests bring a trusted interpreter copy instead of weakening the confinement | Verification gap | __tests__/scripts/writer-fence.test.ts | |
| 95 | The same rule recurred the next day inside a confined agent: a test written beside the helper that enforced it read a process id with a plain number conversion, the file was empty because the confinement rightly refused the launcher the test had built, and the kill went to the test runner's own process group, ending the hook, the version control commit and the agent's commit tool with no output three times; the rule now lives in the test suite's setup, which refuses a signal to the caller's group, to init, or to every process before any signal is sent | Verification gap | __tests__/scripts/kill-guard.test.ts | |
| 96 | A restore check passed on the developer's git release and failed on the runner's: a bundle holding only one branch carries no HEAD, one release adopts the lone branch when cloning it and a newer one leaves an unborn default branch, so main sat red for days on a case nobody could reproduce; the fix restores the bundle's main when HEAD is unborn, and the test forces the newer release's outcome by hand instead of trusting the version that happens to run it | External behavior | __tests__/scripts/collective-restore-drill.test.ts | |
| 97 | Every public page shipped its content inside a hidden streaming block that a script swaps into place after load, because a route-wide loading boundary that rendered nothing wrapped them all, and React moves a large finished boundary out of line even on a prerendered page; browsers showed the pages and another company's AI read the blog as empty, while a link count over the raw HTML called it healthy | Verification gap | __tests__/structure/site-content-in-html.test.ts |
Download: failure-corpus.json, failure-corpus.csv
What changed after a failure?
Each incident below follows one chain: failure, observation, change, evidence and limit. In the first three, an agent or the harness around it failed. The fourth is an ordinary software defect, kept because its check is the pattern the others follow.
The agent finished. The gate threw the work away.
- Failure
- An agent seat, one of the standing agent roles that run on a schedule, finished its run, printed its completion token, then added a two-sentence summary after it. The gate looked for the token only in the last 600 bytes of output, found the summary instead, and discarded a complete eight-minute run. The first fix told all the agents to put the token last, so the gate still depended on the agent obeying.
- Observation
- The run's own structured result said it had ended normally, well inside its turn limit, while the gate filed it as unfinished.
- Change
- The gate now reads the last 25 lines of the output, capped at 8,000 bytes, instead of a fixed 600-byte window. Six weeks later the same class discarded a finished weekly report, because the agent wrote the token into the report file instead of its final message. The token then stopped being the gate: it is recorded as the agent's claim, and a run that ended normally goes on to the guards, which decide.
- Evidence
- For the first fix, the internal lesson records a test that lifts the gate's shell pipeline out of the workflow action and runs it against a reconstruction of the lost run, with a mid-report mention and a truly cut-off result as cases it must still refuse. The public corpus names the workflow action.
- Limit
- When a fix has an instruction half and a mechanism half, only the mechanism held. A reconstruction is not a replay of the original run.
The refusal worked. Its explanation failed.
- Failure
- The agent reported a clean guard run. The guard refused the work, but the outcome did not name the reason and the alarm did not understand the completion state. The site suite and build behind the failing guard had not run.
- Observation
- The agent's closing claim and the guard's failure output disagreed. The work was discarded. Ten alarms across five seats in two weeks, this one included, had been filed the same way, as unexplained, each quoting the agent's claim as the only evidence.
- Change
- The outcome now names the refusal and the stage that refused. The alarm separates the agent's claim from the guard's measurement. The workflow test checks that enumerated literal outcome values, excluding skip, have explicit alarm branches. Alarm tests feed its shell code controlled inputs and check what it prints. Neither test reads a live run.
- Evidence
- The published incident record shows selected correction diffs, paired input and outcome fields, and a test excerpt. Its historical test record reports removing the alarm's branch, regenerating the workflows and watching the check fail.
- Limit
- Both paired cases end in refusal. The corrected case names why. These are published excerpts from private operational records, not a fresh production run. The claim register's version-one verifier does not scan free-form report prose.
The published outcome fields, before and after the correction. Both runs are refused.
Before: the row says only that the run was refused.
{
"outcome": "rejected"
}After: the row names the reason and the stage that refused.
{
"outcome": "rejected",
"reason": "guard-refused",
"refused_by": "caller"
}The prompt named a command the agent could not run.
- Failure
- A prompt fix told the content agent to check its draft with a command that opens a socket. The agent runs inside a fence, a sandbox that limits what it can touch, and the fence refused the socket the command opens, so the one command the fix consisted of could not run. The agent rebuilt the tool by hand, polled a minutes-long test run five times and hit its turn cap with a publishable draft that never reached the guards.
- Observation
- The run was filed as a failure at its turn limit. Reading what the agent did showed the refused command and the turns spent working around it.
- Change
- The prompt now names an in-process form of the command, which opened no socket when measured on a developer machine. A test refuses the socket-opening form in any seat prompt and stays quiet on the safe form and on prose that explains the difference. A second test flags known direct repository-based mkdtemp patterns in scanned TypeScript test files; it does not resolve aliases or relative paths.
- Evidence
- The public corpus names the prompt test. No run of the new form under the fence is recorded.
- Limit
- The test reads prompts for one known command shape. It does not run a seat's commands under the fence, so a different command the fence refuses would pass it.
The log that lied.
- Failure
- The API told callers the requested role or city did not exist while its own log recorded a bad request, on the rows a paying customer reads in the log explorer. The whole suite passed for a month: each test checked one side, and none checked that the two agreed.
- Observation
- Reading the production log table against the responses the API really returned.
- Change
- The source check compares literal logged error statuses with the canonical error registry. It makes a specific disagreement detectable before work lands.
- Evidence
- The public account describes putting the defect back and watching the check fail. The corpus names the source check.
- Limit
- A source check is not a comparison of a production response and its correlated log entry. That would require observations of the same real operation.
Where does harness engineering fail?
In this record, harness engineering failed when an agent could rewrite its later checks, when a protective check took the monitoring down and when an exemption outlived the fact behind it. It also cannot show that a changed instruction changed how often an agent errs.
The agent could rewrite its later checks.
Bash read the workflow step from disk as it ran. Checks placed after the agent in that same script were lines the agent could rewrite before Bash reached them. The correction puts the step in one brace group ending in exit, so Bash parses the compound command before executing it. The named test has a child process rewrite the rest of a script while Bash reads it: the unwrapped step runs the rewrite, and the wrapped step runs what was written. It also reads the workflow files and rejects an agent step left outside the wrapper, apart from two named probe steps that stay unwrapped on purpose.
This closes that specific script-reading path. A brace group is not a complete security boundary or proof of general tamper resistance. The public record does not claim a malicious production exploit. The general rule: a check an agent can write to is not independent. It runs where the agent has no write access, or from bytes held from before the agent started, as the readers after the agent now do. Waiting for the agent's processes to end is not enough, because what it wrote earlier is still there.
Read the public script-rewrite lessonA protective check took the monitoring down.
A publication guard discarded the daily report because of a long number in third-party text. The lesson reports two failed daily runs and pulse and improvement-record artifacts that had gone 51 hours stale. It does not establish the outage duration. Its alarm, repair trigger and watcher were inside or beside the failing system and could not restore it. The response redacts free text at one boundary, prints the generator's real error, checks report age from outside the job and starts a repair after a failed run. The named test feeds the redactor token shapes, long numbers and very large words, and checks that secrets are masked, ordinary text survives and the scan stays fast. It does not run the report or its repair.
The record describes countermeasures, not a demonstrated autonomous recovery. A checker must distinguish an observed defect from an inability to look. A blocker that rejects healthy work has failed too.
Read the public monitoring lessonA check can go stale.
A test that guarded dynamic routes carried an exemption for one route, written down as a negligible leak. Nobody measured it again. Measured later, any bogus name on that route answered with a page and was cached for 90 days, one entry per crawled URL. The exemption was the record of what someone believed on the day they wrote it. The test now accepts only the setting that closes the leak, and this route's exemption was put back with a measured reason, a deploy that ran out of disk when all the pages were built ahead of time, and a routing-layer check that answers before the page renders and before the cache. Checks are also turned down: a visual regression dimension was evaluated and dropped as brittle while it was being built, and its number was never reused.
The record does not yet measure what each check costs in time, and no schedule asks whether each check still earns its place. A check is retired when someone notices it is wrong or too costly, not by a rule.
Read the public exemption lessonThese are one organization's detected and recorded failures. The classifications are the operator's own. The corpus cannot estimate unseen failures or industry prevalence. File existence, behavior on fixtures, production behavior and recurrence prevention are different claims. Private source paths make countermeasures specific; they do not make the underlying private records independently inspectable.
A stored lesson and an existing check do not establish fewer recurrences. As of , the improvement record linked 9 of its 23 incidents to an earlier incident as a recurrence. That is a count, not a rate: 0 of 20 failure families were marked as having a computable opportunity rule. The planned confirmatory window ended on , and it targets publication on . This page has not verified that the launch prerequisites were satisfied or that confirmatory collection occurred. This page reports none of its results. Read the recurrence preregistration.
Where this is going
The goal below is a target. The record above describes particular agent workflows and checks.
The goal is for the Machine, the team of AI agents working on Orbyt Labs, to run on its own without error: each failure is caught by the Machine itself, and a person touches it only through recorded commands and directives. Without error means no error goes unseen or unhandled, not that none occurs.
The record describes particular checks and their evidence, and a rule that keeps agents working unattended from landing a change to what judges them. The next work extends it, from checks that catch a known defect toward measured evals with repeated trials, tests on variants a check has never seen, and a record of what each check costs and when it should go. Features will be added toward the goal. Features not evidenced on this page are outside its claims.
Writing on harness engineering
The Blueprint for the 100x Developer
The boundary between agents and the checks that judge them, and the constrained audit procedure documented for it.
Nobody Asked My Agents to Add These Checks. Not Even Me.
Checks the agents proposed themselves, under one rule: when something slips through, build the check that would have caught it.
Our Cleanup Undid a Developer's Work
A guard's cleanup reverted three concurrent edits, and the registered artifact snapshot that replaced it.
My Tests Passed. The Log Was Wrong.
The response and log disagreement, and a proposed drill that checks the relationship.
84 Ways to Tell Me I'm Wrong.
How individual audit dimensions were earned, with failing output and stated limits.
Codex Accuses. Claude Convicts.
Agents earn authority by passing deterministic checks, and an outside model found the agents could edit those checks.
Read further
- Autonomous operation
A Methods page: how far the Machine is from running on its own, measured.
- Kill switch
A Methods page: what stops the Machine, how fast, and what each stop does not stop.
- Glossary: harness engineering
The definition and its external attribution.
- How we build
The working system and its public receipts.
- The harness timeline
Dated test and audit counts from committed history.
- Audit dimension guide
What each check covers, when it was added and who started it.
- The public failure corpus
Detected incidents, the operator's classifications and named countermeasures.
- The guard field note
Measured guard scripts, blocking checks and quiet-case assertions.
- How the preprint counted its failures
An earlier count of the same corpus, and the rule that a control is live only when its refusal has been observed.
- Synthetic personas
The companion method: how agent drafts are held against review lenses, and where that falls short.
- Governing an Agent Leadership Team: Guardrail, Kill-Switch, and Liveness Patterns from a One-Human Autonomous Organization (opens in a new tab)
The governance preprint and its DOI. Self-published, not peer reviewed.
- Anthropic: Demystifying evals for AI agents (opens in a new tab)
Cited for the field's definition of the agent harness and for running repeated trials. It does not discuss this page.
- Anthropic: Effective harnesses for long-running agents (opens in a new tab)
Cited for the wider sense of harness work: how an agent's work survives across sessions. It does not discuss this page.
- Mitchell Hashimoto: My AI Adoption Journey (opens in a new tab)
February 2026. The account credited above.
Cite this page
Cite this reference for the practice and follow its sources for the underlying measurements.
Bartak, J. (2026). Harness engineering. Orbyt Labs Research. https://