78 Ways the Agents Were Confidently Wrong

The data.

How it was measured.

A census of one corpus, counted by parsing the numbered entries of the failure log on the date shown, not by recalling how many there are.

The silent-failure count matches entries describing a failure that produced no error, no exception and no red test: a green suite, an exit code of zero, a report rendering clean, or a job concluding success while performing no work. From 2.0.0 the classifier is committed code (SILENT in scripts/refresh-research-notes.ts) rather than a hand read, so the number is reproducible and its false negatives are inspectable.

The countermeasure count is the repository's own failure corpus (public/failure-corpus.json, field mechanized_count): an entry is mechanized only when the check, test or guard it names exists in the tree at generation time. Version 1.0.0 counted entries that named a guard in prose; from 2.0.0 the count is a checkable fact about the tree, and it is a lower bound on remediation because some failures were fixed by deleting the capability rather than guarding it.

Both counts come from text matching over the corpus and are reported as such. They classify how a failure was DESCRIBED, which is a proxy for how it behaved.

The corpus is append-only and each entry carries its own receipt: the command, the log line, or the query that established it.

What this does not show.

• These are the failures that were CAUGHT and written down, so the share published here is the silent share among LOGGED failures. The rate across all failures, noticed or not, is unknown, and this corpus cannot bound it in either direction: a failure nobody noticed is not necessarily one that raised no error, because it may have errored somewhere nobody was reading, and a corpus that only holds what was found says nothing about the shape of what was not.
• Text matching classifies prose, not behaviour. An entry that fails to use the word silent is counted as not silent even where the failure was.
• One organization, one codebase, one set of tools. Nothing here establishes that these failure shapes generalize.
• A guard existing is not a guard working. This project separately requires every detector to prove it can stay quiet on a clean input, which is a different measurement published separately.
• No trend is claimed. The changelog carries every refresh, but a rising count is a growing log, not a rising failure rate.

Revisions.

This URL is permanent. When the data is refreshed the version bumps and a row lands here, so a citation made today still resolves to the finding it cited.

Back to all research