Two orange boxy robots, one pointing and one holding a magnifying glass, examine a silver machine of stacked glowing blue and violet circuit layers as a person at monitors watches.

Field note. Version 2.23.0

96 Ways the Agents Were Confidently Wrong

Of 96 distinct engineering failures logged by an AI-run organization as of Oct, 7 2026, 68 of 96 (71%) raised no error at all and were found only because someone checked an artifact directly, and 72 of 96 (75%) have a named automated countermeasure that exists in the repository.

Collected Aug, 27 2026 to Oct, 7 2026. Sample: 96 logged failures in one repository.

The data.

  • Raised no error (silent)68 of 96
  • Have a countermeasure in the tree72 of 96
Of 96 logged failures, 68 of 96 (71%) raised no error at all, and 72 of 96 (75%) now have a countermeasure that exists in the tree.
Measured on the Orbyt Labs repository, Oct, 7 2026. Each figure is produced by the command in its source column.
MetricValueHow it is counted
Distinct failures logged96docs/lessons-learned.md, numbered entries
Failures that raised NO error (silent)68 of 96 (71%)SILENT in scripts/refresh-research-notes.ts over each entry
Failures with a countermeasure that exists in the tree72 of 96 (75%)public/failure-corpus.json, field `mechanized_count`
Automated guards now in the repository118ls scripts/check-*.cjs | wc -l
Average length of a logged failure491 wordsmean word count across the entries

How it was measured.

A census of one corpus, counted by parsing the numbered entries of the failure log on the date shown, not by recalling how many there are.

The silent-failure count matches entries describing a failure that produced no error, no exception and no red test: a green suite, an exit code of zero, a report rendering clean, or a job concluding success while performing no work. From 2.0.0 the classifier is committed code (SILENT in scripts/refresh-research-notes.ts) rather than a hand read, so the number is reproducible and its false negatives are inspectable.

The countermeasure count is the repository's own failure corpus (public/failure-corpus.json, field mechanized_count): an entry is mechanized only when the check, test or guard it names exists in the tree at generation time. Version 1.0.0 counted entries that named a guard in prose; from 2.0.0 the count is a checkable fact about the tree, and it is a lower bound on remediation because some failures were fixed by deleting the capability rather than guarding it.

Both counts come from text matching over the corpus and are reported as such. They classify how a failure was DESCRIBED, which is a proxy for how it behaved.

The corpus is append-only and each entry carries its own receipt: the command, the log line, or the query that established it.

What this does not show.

  • These are the failures that were CAUGHT and written down, so the share published here is the silent share among LOGGED failures. The rate across all failures, noticed or not, is unknown, and this corpus cannot bound it in either direction: a failure nobody noticed is not necessarily one that raised no error, because it may have errored somewhere nobody was reading, and a corpus that only holds what was found says nothing about the shape of what was not.
  • Text matching classifies prose, not behaviour. An entry that fails to use the word silent is counted as not silent even where the failure was.
  • One organization, one codebase, one set of tools. Nothing here establishes that these failure shapes generalize.
  • A guard existing is not a guard working. This project separately requires every detector to prove it can stay quiet on a clean input, which is a different measurement published separately.
  • No trend is claimed. The changelog carries every refresh, but a rising count is a growing log, not a rising failure rate.

Revisions.

This URL is permanent. This table records version numbers and changes.

VersionDateChange
2.23.010-07-2026Refresh: failures 91 -> 96; silent 64 of 91 (70%) -> 68 of 96 (71%); mechanized 67 of 91 (74%) -> 72 of 96 (75%); avgWords 493 words -> 491 words.
2.22.009-26-2026Refresh: avgWords 491 words -> 493 words.
2.21.009-25-2026Refresh: failures 90 -> 91; silent 64 of 90 (71%) -> 64 of 91 (70%); mechanized 66 of 90 (73%) -> 67 of 91 (74%); avgWords 495 words -> 491 words.
2.20.009-24-2026Refresh: avgWords 492 words -> 495 words.
2.19.009-23-2026Refresh: avgWords 491 words -> 492 words.
2.18.009-23-2026Refresh: failures 89 -> 90; silent 63 of 89 (71%) -> 64 of 90 (71%); mechanized 65 of 89 (73%) -> 66 of 90 (73%); avgWords 486 words -> 491 words.
2.17.009-23-2026Refresh: avgWords 485 words -> 486 words.
2.16.009-23-2026Refresh: failures 88 -> 89; silent 63 of 88 (72%) -> 63 of 89 (71%); mechanized 64 of 88 (73%) -> 65 of 89 (73%).
2.15.009-23-2026Refresh: guards 117 -> 118; avgWords 483 words -> 485 words.
2.14.009-23-2026Refresh: guards 116 -> 117; avgWords 482 words -> 483 words.
2.13.009-23-2026Refresh: failures 87 -> 88; silent 62 of 87 (71%) -> 63 of 88 (72%); mechanized 63 of 87 (72%) -> 64 of 88 (73%); avgWords 481 words -> 482 words.
2.12.009-23-2026Refresh: failures 86 -> 87; silent 62 of 86 (72%) -> 62 of 87 (71%); mechanized 62 of 86 (72%) -> 63 of 87 (72%); avgWords 482 words -> 481 words.
2.11.009-23-2026Refresh: guards 115 -> 116.
2.10.009-23-2026Refresh: guards 114 -> 115.
2.9.009-23-2026Refresh: failures 85 -> 86; silent 62 of 85 (73%) -> 62 of 86 (72%); mechanized 61 of 85 (72%) -> 62 of 86 (72%).
2.8.009-22-2026Refresh: avgWords 477 words -> 482 words.
2.7.009-22-2026Refresh: failures 78 -> 85; silent 56 of 78 (72%) -> 62 of 85 (73%); mechanized 54 of 78 (69%) -> 61 of 85 (72%); avgWords 443 words -> 477 words.
2.6.009-17-2026Refresh: failures 75 -> 78; silent 53 of 75 (71%) -> 56 of 78 (72%); mechanized 51 of 75 (68%) -> 54 of 78 (69%); guards 113 -> 114; avgWords 422 words -> 443 words.
2.5.009-13-2026Refresh: failures 74 -> 75; silent 52 of 74 (70%) -> 53 of 75 (71%); mechanized 50 of 74 (68%) -> 51 of 75 (68%); avgWords 416 words -> 422 words.
2.4.009-13-2026Refresh: failures 73 -> 74; silent 51 of 73 (70%) -> 52 of 74 (70%); mechanized 49 of 73 (67%) -> 50 of 74 (68%).
2.3.009-12-2026Refresh: avgWords 413 words -> 416 words.
2.2.009-12-2026Refresh: failures 72 -> 73; silent 50 of 72 (69%) -> 51 of 73 (70%); mechanized 48 of 72 (67%) -> 49 of 73 (67%); avgWords 410 words -> 413 words.
2.1.009-12-2026Refresh: avgWords 406 words -> 410 words.
2.0.009-11-2026Method change and refresh: failures 59 -> 72; silent 34 of 59 (58%) -> 50 of 72 (69%); mechanized 37 of 59 (63%) -> 48 of 72 (67%); guards 105 -> 113; avgWords 379 words -> 406 words. The silent classifier is now committed code (SILENT) rather than a hand read, and the countermeasure count is the failure corpus's mechanized_count, an existence check against the tree rather than a mention in prose. The title moves with the count.
1.0.008-27-2026First publication.

More from Research.

The other measurements from the same repository, and the papers they sit beside.

Cite this.

Bartak, J. (2026). 96 Ways the Agents Were Confidently Wrong. Orbyt Labs Research, version 2.23.0. https://www.orbytlabs.ai/research/confidently-wrong

CC BY 4.0. Reuse it with attribution.

Back to all research