{ "name": "Independent coding of the Orbyt Failure Corpus, as the governance paper read it", "url": "https://www.orbytlabs.ai/research/agent-leadership-governance/independent-coding-2026-09-13.json", "what": "A second, independent rater classified every item of the frozen study input of the paper 'Governing an Agent Leadership Team' into the dataset's three-class taxonomy, blind to the published class. This file holds the instructions verbatim, the rater's full coding with a reason per item, the agreement with the published coding, Cohen's kappa with its formula, the confusion matrix, and every disagreement.", "caveat": "One independent rater coded the frozen corpus from titles alone. It is a language model, not the one that produced the published coding. This is a check on one rater's coding, not peer review. It says nothing about failures the corpus never logged. Agreement between two model raters is not agreement with a human.", "license": "CC BY 4.0", "cite_as": "Orbyt Labs. Independent coding of the Orbyt Failure Corpus as read on 2026-09-02. Rated 2026-09-13 by gpt-6-astra through the OpenAI Codex CLI. https://www.orbytlabs.ai/research/agent-leadership-governance/independent-coding-2026-09-13.json", "generated_at": "2026-09-13", "method": { "date": "2026-09-13", "rater": { "tool": "OpenAI Codex CLI v0.153.2", "model": "gpt-6-astra", "provider": "openai", "reasoning_effort": "xhigh", "command": "codex exec --sandbox read-only --skip-git-repo-check -C ", "prompt_delivery": "the instructions below, on stdin", "session_id": "01a09d7e-f25c-7352-8743-17703a04910d", "tokens_used": 14878 }, "original_coding": "The published `kind` on each item of the frozen file: one rater, the agent that operates the organization, coding from the full internal forensics. The paper states this limitation.", "blinding": "The rater received each item's id and public title only. The published kind, the item's date, its countermeasure path and the internal forensics were withheld. The working directory held only the instructions file and the id-and-title list. The sandbox was read-only.", "sample": { "description": "The whole frozen study input the paper read, every item, no sampling.", "repo_path": "public/research/agent-leadership-governance/failure-corpus-2026-09-02.json", "served_path": "/research/agent-leadership-governance/failure-corpus-2026-09-02.json", "sha256": "efbe0e55e43415c514e187c81b1ab44ab36487dba993860bb7c37e290148f070", "count": 66, "fields_shown_to_rater": [ "id", "title" ] }, "kind_definitions": { "external-behavior": "An external platform, client, or dependency behaved differently than documented or assumed.", "verification-gap": "A test, guard, gate, or metric that could not fail, could not see, or measured a proxy for the thing it claimed to measure.", "operator-process": "A discipline failure in how the AI operator built or maintained the system: stale documents, unswept invariants, contracts with one side missing." }, "instructions": "# Independent coding of an engineering failure corpus\n\nYou are an independent rater. Classify each of the 66 records below into exactly one of three classes. You are given only each record's id and its public title. You have not been given, and must not try to find, any prior classification of these records.\n\n## The three classes, quoted verbatim from the taxonomy that ships with the dataset\n\n- external-behavior: \"An external platform, client, or dependency behaved differently than documented or assumed.\"\n- verification-gap: \"A test, guard, gate, or metric that could not fail, could not see, or measured a proxy for the thing it claimed to measure.\"\n- operator-process: \"A discipline failure in how the AI operator built or maintained the system: stale documents, unswept invariants, contracts with one side missing.\"\n\n## Rules\n\n1. Use the title alone. Do not search the web, do not open any other file, and do not infer a class from the id.\n2. Assign every record exactly one class. Do not skip a record. If a title fits two classes, choose the class that names the ROOT of the failure and say so in the reason.\n3. Give one sentence of reason per record, in your own words, naming what in the title decided the class.\n4. Reply with a single JSON array and nothing after it: exactly 66 elements, one per id, in ascending id order, each of the shape {\"id\": , \"kind\": \"\", \"one_sentence_reason\": \"\"}. A markdown fence around the array is acceptable. No prose after the array.\n\n## The records (id and title only; the same list is in items.json beside this file)\n\n[\n {\n \"id\": 1,\n \"title\": \"A mobile browser can suspend a tab without releasing its Web Locks, deadlocking auth refresh\"\n },\n {\n \"id\": 2,\n \"title\": \"Realtime sync can echo a write back into itself unless transient UI state stays out of the database\"\n },\n {\n \"id\": 3,\n \"title\": \"High-frequency writes to a replicated column can destabilize the replication layer itself\"\n },\n {\n \"id\": 4,\n \"title\": \"A cleared cache with a surviving session is a distinct state; UI must wait for hydration to complete\"\n },\n {\n \"id\": 5,\n \"title\": \"A managed provider's default email path cannot be assumed to deliver; route through one you can observe\"\n },\n {\n \"id\": 6,\n \"title\": \"Unsafe HTML injection requires an explicit, same-line trust annotation to pass review\"\n },\n {\n \"id\": 7,\n \"title\": \"A module whose internal name diverges from its export breaks tooling that assumes they match\"\n },\n {\n \"id\": 8,\n \"title\": \"After a rename, the old name is a liability: redirect it once, then never reference it again\"\n },\n {\n \"id\": 9,\n \"title\": \"A date test that mocks the clock in a different timezone convention than the code parses fails only west of UTC\"\n },\n {\n \"id\": 10,\n \"title\": \"A client library that auto-deserializes on read breaks every consumer that stored a string, and only on cache hit\"\n },\n {\n \"id\": 11,\n \"title\": \"A programmatic page matrix needs a bounded kept-set imported by both the builder and the sitemap\"\n },\n {\n \"id\": 12,\n \"title\": \"An internal spec date is not a protocol version; clients hard-reject unknown values\"\n },\n {\n \"id\": 13,\n \"title\": \"A redirect parameter is a contract between writer and reader; a silent fallback hides the mismatch\"\n },\n {\n \"id\": 14,\n \"title\": \"Hiding overflow on one axis creates a scroll container on the other and can trap wheel events\"\n },\n {\n \"id\": 15,\n \"title\": \"A client-reachable module that statically imports a heavy data source ships it to every page that renders it\"\n },\n {\n \"id\": 16,\n \"title\": \"A test that derives its expected value from the code under test can never fail\"\n },\n {\n \"id\": 17,\n \"title\": \"A pre-authentication code path needs its own abuse ceiling\"\n },\n {\n \"id\": 18,\n \"title\": \"A table listing's row counts are planner estimates; report a real count or report nothing\"\n },\n {\n \"id\": 19,\n \"title\": \"Email clients default to content-box, so full width plus padding overflows a phone\"\n },\n {\n \"id\": 20,\n \"title\": \"A CI gate that only runs on the day it matters has never actually been tested; force the condition once\"\n },\n {\n \"id\": 21,\n \"title\": \"A convention that has never been executed is not a working convention\"\n },\n {\n \"id\": 22,\n \"title\": \"A build filter that decides inclusion is a detector, and must be proven on the shortest legitimate case\"\n },\n {\n \"id\": 23,\n \"title\": \"Invalid CSS is discarded, not flagged; a broken value silently becomes the property's initial value\"\n },\n {\n \"id\": 24,\n \"title\": \"Read a provider response by scanning for the block you want, never by indexing position zero\"\n },\n {\n \"id\": 25,\n \"title\": \"A model's output budget caps reasoning plus text; leave neither half implicit\"\n },\n {\n \"id\": 26,\n \"title\": \"A tool that runs quarterly has no gate between quarters, and its hand-copied config drifts silently\"\n },\n {\n \"id\": 27,\n \"title\": \"A cron reporting success is not healthy; watch duration against its ceiling and reap rows whose process never returned\"\n },\n {\n \"id\": 28,\n \"title\": \"A timeout reports as cancelled, not failed; alarms keyed on failure alone are structurally blind to it\"\n },\n {\n \"id\": 29,\n \"title\": \"A metric measured and then discarded is indistinguishable from a metric never measured\"\n },\n {\n \"id\": 30,\n \"title\": \"A synchronous spawn against an in-process fake server deadlocks, and the hang reads as the child failing\"\n },\n {\n \"id\": 31,\n \"title\": \"A step that discards stderr turns a missing permission into a silently wrong branch\"\n },\n {\n \"id\": 32,\n \"title\": \"A relabeled invariant must sweep every detector that encodes it, in the same commit\"\n },\n {\n \"id\": 33,\n \"title\": \"Checks sharing one rate-limit window flake each other; absorb exactly one window or serialize\"\n },\n {\n \"id\": 34,\n \"title\": \"A fix committed but not pushed does not exist, and a guard grading the working tree will certify it as live\"\n },\n {\n \"id\": 35,\n \"title\": \"An instrument's failure paths must render as not observed, never as clean\"\n },\n {\n \"id\": 36,\n \"title\": \"A lesson mechanized in the canonical pipeline does not exist for the surfaces that bypass it\"\n },\n {\n \"id\": 37,\n \"title\": \"A selection rule optimizes its proxy; verify what exists against measured demand, not against the code\"\n },\n {\n \"id\": 38,\n \"title\": \"A sync rule that tests equality can never be satisfied by the thing it exists to allow; record provenance instead\"\n },\n {\n \"id\": 39,\n \"title\": \"Never validate a filter against data that filter produced\"\n },\n {\n \"id\": 40,\n \"title\": \"A gate whose correctness rests on the thing it is gating is not a gate\"\n },\n {\n \"id\": 41,\n \"title\": \"A job cancelled with zero steps was never handed to a runner; an alarm inside the job cannot see the job never starting\"\n },\n {\n \"id\": 42,\n \"title\": \"Read a monitor's freshness the way the monitor is actually fed\"\n },\n {\n \"id\": 43,\n \"title\": \"A stale document reads as instructions; when an invariant reverses, sweep the documents that assert it\"\n },\n {\n \"id\": 44,\n \"title\": \"A sandbox that cannot start reports success, and a fence can apply halfway\"\n },\n {\n \"id\": 45,\n \"title\": \"A confinement lane is what an agent is instructed to write, not what gets committed\"\n },\n {\n \"id\": 46,\n \"title\": \"A guard's exit code is not its verdict; report-only and strict-gated checks exit zero while reporting\"\n },\n {\n \"id\": 47,\n \"title\": \"A gate that reads a tool's printed summary is reading a rendering, and colour in CI can leave it with no way to fail\"\n },\n {\n \"id\": 48,\n \"title\": \"Permissive access policies combine with OR, so one globally true policy defeats every narrow policy beside it, and two individually correct guards can still leave the class unowned\"\n },\n {\n \"id\": 49,\n \"title\": \"A generated file that measures the repository containing it can never be correct at the moment it is committed, and a drift guard that performs no regeneration of its own reports PASS on a precondition it never established\"\n },\n {\n \"id\": 50,\n \"title\": \"A shallow repository clone makes the version-control history answer confidently and wrongly, so a generator that trusts it publishes a false freshness claim with no error anywhere\"\n },\n {\n \"id\": 51,\n \"title\": \"A privileged key does not bypass a control the upstream service enforces, and charging that service's refusal to a user's abuse budget locks out the innocent\"\n },\n {\n \"id\": 52,\n \"title\": \"An upstream service reports a downstream failure in its own vocabulary, so the error names a cause that is not the real one, and four consecutive fixes were aimed at the wrong side of the wire\"\n },\n {\n \"id\": 53,\n \"title\": \"Internal documentation is not a wire contract: a set of event names transcribed from reviewed internal docs matched nothing the sender actually emits, and a single observed value disagreeing with a document invalidates the whole document rather than the one row it lands on\"\n },\n {\n \"id\": 54,\n \"title\": \"A monitoring query that samples without filtering reports the absence of the needle as the absence of the problem, and sorting newest-first does not fix it; a reported count equal to the page size is never a measurement\"\n },\n {\n \"id\": 55,\n \"title\": \"A dependency's implicit fetch option is a route-classification decision, and it can silently repeal a page's own segment config, turning a declared 404 into a streamed 200 soft-404 that only the deployed build exhibits\"\n },\n {\n \"id\": 56,\n \"title\": \"An allowlist entry is a claim about production that decays silently, and a guard checking whether a value is declared cannot go loud about the value that actually costs money\"\n },\n {\n \"id\": 57,\n \"title\": \"A verification that normalizes its input passes on a value the real consumer will reject, so an instrument must read its subject exactly the way the consumer reads it\"\n },\n {\n \"id\": 58,\n \"title\": \"A cost measurement is valid only for the resource it measured on the machine that binds it, and a local build stops one whole stage before the one that fails\"\n },\n {\n \"id\": 59,\n \"title\": \"A field that may contain markup is safe only where it is rendered as markup, and the sink no human ever looks at is the one that keeps shipping the defect\"\n },\n {\n \"id\": 60,\n \"title\": \"A proximity window measured in lines stops measuring anything once a generator writes the file, and a framing verb inside a citation is not a claim of affiliation\"\n },\n {\n \"id\": 61,\n \"title\": \"A test that spawns an entrypoint aware of its build environment inherits the live reporting channel, so a suite can overwrite the run's own verdict and the last write wins\"\n },\n {\n \"id\": 62,\n \"title\": \"A hand edit to a generated file is one run from silent reversion, and a suite that grades the file instead of a regeneration will never see it\"\n },\n {\n \"id\": 63,\n \"title\": \"A guard that reads the wall clock is vacuous at the wrong hour, so deleting the fix it protects can leave it green\"\n },\n {\n \"id\": 64,\n \"title\": \"A corpus gated on the wrong date field publishes a placeholder before the enriched version can exist, and an exemption list written the day a feature lands is the tell\"\n },\n {\n \"id\": 65,\n \"title\": \"A shell capture of a non-string value from a runtime that colorizes its output returns ANSI escape bytes, and the environment that does it is the automation harness rather than a human terminal\"\n },\n {\n \"id\": 66,\n \"title\": \"Calendar month arithmetic overflows into the following month, and a test that computes its expectation the same way agrees with the defect instead of catching it\"\n }\n]\n", "raw_output_sha256": "d94f3968421d6ee4350ac8f8ec9c872540662e043ec7fc5bc2b2a0f32b7f3887" }, "results": { "n": 66, "agreed": 52, "agreement_rate": 0.7878787878787878, "kappa": { "value": 0.668103448275862, "observed_agreement_po": 0.7878787878787878, "expected_agreement_pe": 0.3608815426997245, "formula": "kappa = (po - pe) / (1 - pe), where po = agreed / n is the observed agreement and pe = sum over the three kinds of (paper row total x independent column total) / n^2 is the agreement expected by chance from each rater's marginal class counts." }, "confusion_matrix": { "rows_are": "paper (the published kind)", "columns_are": "independent (the second rater's kind)", "labels": [ "external-behavior", "verification-gap", "operator-process" ], "counts": [ [ 13, 1, 1 ], [ 3, 25, 5 ], [ 1, 3, 14 ] ], "row_totals": [ 15, 33, 18 ], "column_totals": [ 17, 29, 20 ] }, "class_counts": { "paper": { "external-behavior": 15, "verification-gap": 33, "operator-process": 18 }, "independent": { "external-behavior": 17, "verification-gap": 29, "operator-process": 20 } }, "verification_gap_share": { "paper": { "count": 33, "of": 66 }, "independent": { "count": 29, "of": 66 } }, "disagreement_count": 14 }, "disagreements": [ { "id": 2, "title": "Realtime sync can echo a write back into itself unless transient UI state stays out of the database", "paper": "external-behavior", "independent": "operator-process", "reason": "The root is persisting transient UI state in the database, which lets realtime synchronization feed a write back into itself." }, { "id": 21, "title": "A convention that has never been executed is not a working convention", "paper": "verification-gap", "independent": "operator-process", "reason": "The title identifies reliance on a convention that has never been carried out, making the failure one of execution discipline." }, { "id": 26, "title": "A tool that runs quarterly has no gate between quarters, and its hand-copied config drifts silently", "paper": "verification-gap", "independent": "operator-process", "reason": "The root is maintaining a rarely executed tool through hand-copied configuration without upkeep between runs, allowing silent drift." }, { "id": 33, "title": "Checks sharing one rate-limit window flake each other; absorb exactly one window or serialize", "paper": "external-behavior", "independent": "verification-gap", "reason": "The checks consume one another's rate-limit allowance, so their failures reflect interference between checks instead of independent evidence about the system." }, { "id": 34, "title": "A fix committed but not pushed does not exist, and a guard grading the working tree will certify it as live", "paper": "verification-gap", "independent": "operator-process", "reason": "The root is leaving the fix unpushed and therefore undelivered, while the working-tree guard conceals that incomplete delivery." }, { "id": 37, "title": "A selection rule optimizes its proxy; verify what exists against measured demand, not against the code", "paper": "operator-process", "independent": "verification-gap", "reason": "The selection rule is judged against its own proxy instead of measured demand, so verification cannot establish whether the resulting selection serves the intended need." }, { "id": 38, "title": "A sync rule that tests equality can never be satisfied by the thing it exists to allow; record provenance instead", "paper": "operator-process", "independent": "verification-gap", "reason": "The sync guard's equality condition excludes the very case it should permit, making its predicate incapable of recognizing valid synchronization." }, { "id": 44, "title": "A sandbox that cannot start reports success, and a fence can apply halfway", "paper": "verification-gap", "independent": "external-behavior", "reason": "The root is sandbox behavior that reports success despite startup failure and can apply confinement only partially, violating assumed execution guarantees." }, { "id": 46, "title": "A guard's exit code is not its verdict; report-only and strict-gated checks exit zero while reporting", "paper": "operator-process", "independent": "verification-gap", "reason": "The root is treating a guard's exit code as its verdict even though the guard can report findings while exiting successfully." }, { "id": 48, "title": "Permissive access policies combine with OR, so one globally true policy defeats every narrow policy beside it, and two individually correct guards can still leave the class unowned", "paper": "verification-gap", "independent": "external-behavior", "reason": "The root is the platform's OR combination of permissive policies, which lets a globally true policy defeat narrower restrictions while gaps between guards leave the exposure undetected." }, { "id": 52, "title": "An upstream service reports a downstream failure in its own vocabulary, so the error names a cause that is not the real one, and four consecutive fixes were aimed at the wrong side of the wire", "paper": "operator-process", "independent": "external-behavior", "reason": "The root is an upstream service translating a downstream failure into misleading error vocabulary, which directs subsequent fixes toward the wrong component." }, { "id": 59, "title": "A field that may contain markup is safe only where it is rendered as markup, and the sink no human ever looks at is the one that keeps shipping the defect", "paper": "verification-gap", "independent": "operator-process", "reason": "The root is failing to maintain the markup field's rendering contract across every sink, with the overlooked sink allowing the defect to persist." }, { "id": 62, "title": "A hand edit to a generated file is one run from silent reversion, and a suite that grades the file instead of a regeneration will never see it", "paper": "verification-gap", "independent": "operator-process", "reason": "The root is editing generated output instead of its source of truth, while a suite that skips regeneration conceals the edit's inevitable reversion." }, { "id": 66, "title": "Calendar month arithmetic overflows into the following month, and a test that computes its expectation the same way agrees with the defect instead of catching it", "paper": "verification-gap", "independent": "external-behavior", "reason": "The root is calendar month arithmetic overflowing beyond the intended month, while a test that repeats the same arithmetic merely conceals that unexpected behavior." } ], "ratings": [ { "id": 1, "title": "A mobile browser can suspend a tab without releasing its Web Locks, deadlocking auth refresh", "kind": "external-behavior", "one_sentence_reason": "The mobile browser suspends a tab while retaining its Web Locks, contradicting the assumption that suspension releases locks needed for auth refresh." }, { "id": 2, "title": "Realtime sync can echo a write back into itself unless transient UI state stays out of the database", "kind": "operator-process", "one_sentence_reason": "The root is persisting transient UI state in the database, which lets realtime synchronization feed a write back into itself." }, { "id": 3, "title": "High-frequency writes to a replicated column can destabilize the replication layer itself", "kind": "external-behavior", "one_sentence_reason": "The replication layer becomes unstable under frequent column writes, violating the assumed behavior of that dependency under load." }, { "id": 4, "title": "A cleared cache with a surviving session is a distinct state; UI must wait for hydration to complete", "kind": "operator-process", "one_sentence_reason": "The UI fails to account for a surviving session with an empty cache and renders before hydration has completed." }, { "id": 5, "title": "A managed provider's default email path cannot be assumed to deliver; route through one you can observe", "kind": "external-behavior", "one_sentence_reason": "The root is the managed provider's default email path failing the assumption that messages will arrive, with observability addressing the resulting uncertainty." }, { "id": 6, "title": "Unsafe HTML injection requires an explicit, same-line trust annotation to pass review", "kind": "operator-process", "one_sentence_reason": "The title makes an explicit trust annotation beside unsafe HTML injection a required part of implementation and review discipline." }, { "id": 7, "title": "A module whose internal name diverges from its export breaks tooling that assumes they match", "kind": "operator-process", "one_sentence_reason": "The root is allowing the module's internal name and exported name to diverge, breaking the naming contract used by tooling." }, { "id": 8, "title": "After a rename, the old name is a liability: redirect it once, then never reference it again", "kind": "operator-process", "one_sentence_reason": "The title identifies incomplete rename maintenance, requiring old references to be swept after establishing a redirect." }, { "id": 9, "title": "A date test that mocks the clock in a different timezone convention than the code parses fails only west of UTC", "kind": "verification-gap", "one_sentence_reason": "The test uses a clock mock with timezone conventions inconsistent with the parser, making its result depend on an invalid test setup." }, { "id": 10, "title": "A client library that auto-deserializes on read breaks every consumer that stored a string, and only on cache hit", "kind": "external-behavior", "one_sentence_reason": "The client library automatically deserializes cached values, contradicting consumers' assumption that a stored string will remain a string when read." }, { "id": 11, "title": "A programmatic page matrix needs a bounded kept-set imported by both the builder and the sitemap", "kind": "operator-process", "one_sentence_reason": "The builder and sitemap need one shared bounded inclusion set to maintain their agreement about which pages exist." }, { "id": 12, "title": "An internal spec date is not a protocol version; clients hard-reject unknown values", "kind": "operator-process", "one_sentence_reason": "The root is substituting an internal specification date for a valid protocol version, violating the client-facing contract." }, { "id": 13, "title": "A redirect parameter is a contract between writer and reader; a silent fallback hides the mismatch", "kind": "operator-process", "one_sentence_reason": "The root is disagreement between the redirect parameter's writer and reader, while the silent fallback conceals that broken contract." }, { "id": 14, "title": "Hiding overflow on one axis creates a scroll container on the other and can trap wheel events", "kind": "external-behavior", "one_sentence_reason": "The browser's overflow rules create scrolling behavior on the other axis, contradicting the assumption that hiding one axis affects only that axis." }, { "id": 15, "title": "A client-reachable module that statically imports a heavy data source ships it to every page that renders it", "kind": "operator-process", "one_sentence_reason": "The root is placing a heavy static import in a client-reachable module, causing the dependency to ship with every page that renders it." }, { "id": 16, "title": "A test that derives its expected value from the code under test can never fail", "kind": "verification-gap", "one_sentence_reason": "The test derives its expected value from the implementation itself, eliminating the independent comparison needed to detect a defect." }, { "id": 17, "title": "A pre-authentication code path needs its own abuse ceiling", "kind": "operator-process", "one_sentence_reason": "The title identifies an omitted protection in system construction: a pre-authentication path lacks its own abuse ceiling." }, { "id": 18, "title": "A table listing's row counts are planner estimates; report a real count or report nothing", "kind": "verification-gap", "one_sentence_reason": "The root is presenting planner estimates as actual row counts, substituting a proxy for the quantity the measurement claims to report." }, { "id": 19, "title": "Email clients default to content-box, so full width plus padding overflows a phone", "kind": "external-behavior", "one_sentence_reason": "Email clients use content-box sizing by default, so padding increases the rendered width beyond the assumed full-width boundary." }, { "id": 20, "title": "A CI gate that only runs on the day it matters has never actually been tested; force the condition once", "kind": "verification-gap", "one_sentence_reason": "The CI gate's decisive condition has never been exercised, leaving its ability to detect the relevant failure unverified." }, { "id": 21, "title": "A convention that has never been executed is not a working convention", "kind": "operator-process", "one_sentence_reason": "The title identifies reliance on a convention that has never been carried out, making the failure one of execution discipline." }, { "id": 22, "title": "A build filter that decides inclusion is a detector, and must be proven on the shortest legitimate case", "kind": "verification-gap", "one_sentence_reason": "The inclusion filter lacks evidence that it recognizes the shortest legitimate input, leaving a boundary of its detection capability unchecked." }, { "id": 23, "title": "Invalid CSS is discarded, not flagged; a broken value silently becomes the property's initial value", "kind": "external-behavior", "one_sentence_reason": "CSS processing silently discards an invalid value instead of flagging it, producing fallback behavior that differs from the assumed declaration." }, { "id": 24, "title": "Read a provider response by scanning for the block you want, never by indexing position zero", "kind": "external-behavior", "one_sentence_reason": "The provider can return the desired response block outside position zero, contradicting the consumer's assumed ordering." }, { "id": 25, "title": "A model's output budget caps reasoning plus text; leave neither half implicit", "kind": "external-behavior", "one_sentence_reason": "The model's output limit covers reasoning and visible text together, contradicting an assumption that either component has an independent budget." }, { "id": 26, "title": "A tool that runs quarterly has no gate between quarters, and its hand-copied config drifts silently", "kind": "operator-process", "one_sentence_reason": "The root is maintaining a rarely executed tool through hand-copied configuration without upkeep between runs, allowing silent drift." }, { "id": 27, "title": "A cron reporting success is not healthy; watch duration against its ceiling and reap rows whose process never returned", "kind": "verification-gap", "one_sentence_reason": "Reported cron success is an inadequate proxy for health because it does not establish acceptable duration or identify executions that never returned." }, { "id": 28, "title": "A timeout reports as cancelled, not failed; alarms keyed on failure alone are structurally blind to it", "kind": "verification-gap", "one_sentence_reason": "The root is an alarm that observes only failure status, leaving timeouts invisible when they are classified as cancellations." }, { "id": 29, "title": "A metric measured and then discarded is indistinguishable from a metric never measured", "kind": "verification-gap", "one_sentence_reason": "Discarding a measured metric prevents it from providing any usable evidence about the condition it was meant to observe." }, { "id": 30, "title": "A synchronous spawn against an in-process fake server deadlocks, and the hang reads as the child failing", "kind": "verification-gap", "one_sentence_reason": "The root is a verification harness that blocks its own fake server through synchronous spawning, making the apparent child failure an artifact of the test setup." }, { "id": 31, "title": "A step that discards stderr turns a missing permission into a silently wrong branch", "kind": "verification-gap", "one_sentence_reason": "Discarding stderr hides the permission failure that the step needs to distinguish an error from a valid branching condition." }, { "id": 32, "title": "A relabeled invariant must sweep every detector that encodes it, in the same commit", "kind": "operator-process", "one_sentence_reason": "The title requires an invariant change to propagate to every detector that encodes it, identifying an incomplete maintenance sweep as the failure." }, { "id": 33, "title": "Checks sharing one rate-limit window flake each other; absorb exactly one window or serialize", "kind": "verification-gap", "one_sentence_reason": "The checks consume one another's rate-limit allowance, so their failures reflect interference between checks instead of independent evidence about the system." }, { "id": 34, "title": "A fix committed but not pushed does not exist, and a guard grading the working tree will certify it as live", "kind": "operator-process", "one_sentence_reason": "The root is leaving the fix unpushed and therefore undelivered, while the working-tree guard conceals that incomplete delivery." }, { "id": 35, "title": "An instrument's failure paths must render as not observed, never as clean", "kind": "verification-gap", "one_sentence_reason": "The instrument treats its own inability to observe as a clean result, allowing measurement failure to masquerade as evidence of health." }, { "id": 36, "title": "A lesson mechanized in the canonical pipeline does not exist for the surfaces that bypass it", "kind": "operator-process", "one_sentence_reason": "The root is applying a learned constraint only in the canonical pipeline without carrying it to the surfaces that bypass that pipeline." }, { "id": 37, "title": "A selection rule optimizes its proxy; verify what exists against measured demand, not against the code", "kind": "verification-gap", "one_sentence_reason": "The selection rule is judged against its own proxy instead of measured demand, so verification cannot establish whether the resulting selection serves the intended need." }, { "id": 38, "title": "A sync rule that tests equality can never be satisfied by the thing it exists to allow; record provenance instead", "kind": "verification-gap", "one_sentence_reason": "The sync guard's equality condition excludes the very case it should permit, making its predicate incapable of recognizing valid synchronization." }, { "id": 39, "title": "Never validate a filter against data that filter produced", "kind": "verification-gap", "one_sentence_reason": "Validating a filter using only its own output excludes rejected inputs from scrutiny and makes the evidence circular." }, { "id": 40, "title": "A gate whose correctness rests on the thing it is gating is not a gate", "kind": "verification-gap", "one_sentence_reason": "The gate relies on the correctness of its subject, so it cannot independently establish the condition it claims to enforce." }, { "id": 41, "title": "A job cancelled with zero steps was never handed to a runner; an alarm inside the job cannot see the job never starting", "kind": "verification-gap", "one_sentence_reason": "The root is locating the alarm inside the monitored job, where it cannot execute or observe anything when the job never reaches a runner." }, { "id": 42, "title": "Read a monitor's freshness the way the monitor is actually fed", "kind": "verification-gap", "one_sentence_reason": "The freshness assessment uses assumptions inconsistent with how the monitor receives updates, so it does not reliably measure the monitor's actual freshness." }, { "id": 43, "title": "A stale document reads as instructions; when an invariant reverses, sweep the documents that assert it", "kind": "operator-process", "one_sentence_reason": "Documents retain a reversed invariant and continue directing work according to obsolete instructions because the maintenance sweep was incomplete." }, { "id": 44, "title": "A sandbox that cannot start reports success, and a fence can apply halfway", "kind": "external-behavior", "one_sentence_reason": "The root is sandbox behavior that reports success despite startup failure and can apply confinement only partially, violating assumed execution guarantees." }, { "id": 45, "title": "A confinement lane is what an agent is instructed to write, not what gets committed", "kind": "operator-process", "one_sentence_reason": "The title locates confinement in the scope of the agent's writing instructions, making committed output an insufficient basis for maintaining that operating boundary." }, { "id": 46, "title": "A guard's exit code is not its verdict; report-only and strict-gated checks exit zero while reporting", "kind": "verification-gap", "one_sentence_reason": "The root is treating a guard's exit code as its verdict even though the guard can report findings while exiting successfully." }, { "id": 47, "title": "A gate that reads a tool's printed summary is reading a rendering, and colour in CI can leave it with no way to fail", "kind": "verification-gap", "one_sentence_reason": "The root is a gate parsing presentation text whose CI coloring can defeat detection, leaving the gate unable to fail on reported problems." }, { "id": 48, "title": "Permissive access policies combine with OR, so one globally true policy defeats every narrow policy beside it, and two individually correct guards can still leave the class unowned", "kind": "external-behavior", "one_sentence_reason": "The root is the platform's OR combination of permissive policies, which lets a globally true policy defeat narrower restrictions while gaps between guards leave the exposure undetected." }, { "id": 49, "title": "A generated file that measures the repository containing it can never be correct at the moment it is committed, and a drift guard that performs no regeneration of its own reports PASS on a precondition it never established", "kind": "verification-gap", "one_sentence_reason": "The repository measurement invalidates itself when committed, and the drift guard compounds that verification failure by passing without establishing its regeneration precondition." }, { "id": 50, "title": "A shallow repository clone makes the version-control history answer confidently and wrongly, so a generator that trusts it publishes a false freshness claim with no error anywhere", "kind": "external-behavior", "one_sentence_reason": "The root is version-control history returning misleading answers from a shallow clone without errors, defeating the generator's assumption that those answers establish freshness." }, { "id": 51, "title": "A privileged key does not bypass a control the upstream service enforces, and charging that service's refusal to a user's abuse budget locks out the innocent", "kind": "external-behavior", "one_sentence_reason": "The root is the upstream service enforcing a control despite a privileged key, with the later abuse-budget charge misattributing that unexpected refusal to the user." }, { "id": 52, "title": "An upstream service reports a downstream failure in its own vocabulary, so the error names a cause that is not the real one, and four consecutive fixes were aimed at the wrong side of the wire", "kind": "external-behavior", "one_sentence_reason": "The root is an upstream service translating a downstream failure into misleading error vocabulary, which directs subsequent fixes toward the wrong component." }, { "id": 53, "title": "Internal documentation is not a wire contract: a set of event names transcribed from reviewed internal docs matched nothing the sender actually emits, and a single observed value disagreeing with a document invalidates the whole document rather than the one row it lands on", "kind": "operator-process", "one_sentence_reason": "The root is treating internal documentation as the wire contract and failing to revalidate the entire event-name set after observed behavior contradicts it." }, { "id": 54, "title": "A monitoring query that samples without filtering reports the absence of the needle as the absence of the problem, and sorting newest-first does not fix it; a reported count equal to the page size is never a measurement", "kind": "verification-gap", "one_sentence_reason": "The monitoring query mistakes an unfiltered sample and its page limit for evidence about absence and total count, so it cannot measure the claimed condition." }, { "id": 55, "title": "A dependency's implicit fetch option is a route-classification decision, and it can silently repeal a page's own segment config, turning a declared 404 into a streamed 200 soft-404 that only the deployed build exhibits", "kind": "external-behavior", "one_sentence_reason": "A dependency's implicit fetch option changes deployed route classification and overrides the page's declared behavior, producing an unexpected streamed 200 response." }, { "id": 56, "title": "An allowlist entry is a claim about production that decays silently, and a guard checking whether a value is declared cannot go loud about the value that actually costs money", "kind": "verification-gap", "one_sentence_reason": "The root is a guard checking declared allowlist membership instead of the production value that incurs cost, leaving real spending changes outside its view." }, { "id": 57, "title": "A verification that normalizes its input passes on a value the real consumer will reject, so an instrument must read its subject exactly the way the consumer reads it", "kind": "verification-gap", "one_sentence_reason": "The verifier normalizes input that the consumer reads unchanged, allowing it to approve values the real consumer rejects." }, { "id": 58, "title": "A cost measurement is valid only for the resource it measured on the machine that binds it, and a local build stops one whole stage before the one that fails", "kind": "verification-gap", "one_sentence_reason": "The cost measurement covers the wrong resource or machine, and the local build omits the failing stage, leaving both forms of evidence unable to establish the claimed result." }, { "id": 59, "title": "A field that may contain markup is safe only where it is rendered as markup, and the sink no human ever looks at is the one that keeps shipping the defect", "kind": "operator-process", "one_sentence_reason": "The root is failing to maintain the markup field's rendering contract across every sink, with the overlooked sink allowing the defect to persist." }, { "id": 60, "title": "A proximity window measured in lines stops measuring anything once a generator writes the file, and a framing verb inside a citation is not a claim of affiliation", "kind": "verification-gap", "one_sentence_reason": "The detector uses line proximity and isolated citation wording as proxies that do not establish semantic proximity or an affiliation claim." }, { "id": 61, "title": "A test that spawns an entrypoint aware of its build environment inherits the live reporting channel, so a suite can overwrite the run's own verdict and the last write wins", "kind": "verification-gap", "one_sentence_reason": "The root is a test harness sharing the live reporting channel with spawned entrypoints, allowing test activity to overwrite the verdict it is supposed to establish." }, { "id": 62, "title": "A hand edit to a generated file is one run from silent reversion, and a suite that grades the file instead of a regeneration will never see it", "kind": "operator-process", "one_sentence_reason": "The root is editing generated output instead of its source of truth, while a suite that skips regeneration conceals the edit's inevitable reversion." }, { "id": 63, "title": "A guard that reads the wall clock is vacuous at the wrong hour, so deleting the fix it protects can leave it green", "kind": "verification-gap", "one_sentence_reason": "The guard's wall-clock dependency makes it vacuous outside the relevant hour, so removing the protected fix can still produce a passing result." }, { "id": 64, "title": "A corpus gated on the wrong date field publishes a placeholder before the enriched version can exist, and an exemption list written the day a feature lands is the tell", "kind": "verification-gap", "one_sentence_reason": "The root is a publication gate using the wrong date field as a proxy for enrichment readiness, allowing a placeholder through before an enriched version can exist." }, { "id": 65, "title": "A shell capture of a non-string value from a runtime that colorizes its output returns ANSI escape bytes, and the environment that does it is the automation harness rather than a human terminal", "kind": "external-behavior", "one_sentence_reason": "The runtime emits ANSI color bytes for a non-string value inside the automation harness, contradicting the shell capture's assumption of plain output." }, { "id": 66, "title": "Calendar month arithmetic overflows into the following month, and a test that computes its expectation the same way agrees with the defect instead of catching it", "kind": "external-behavior", "one_sentence_reason": "The root is calendar month arithmetic overflowing beyond the intended month, while a test that repeats the same arithmetic merely conceals that unexpected behavior." } ] }