Did the Machine Learn From Its Last Failure?

The data.

How it was measured.

This preregistration sets the question, design and stop rule before confirmatory collection. This note reports no study result. I will publish results at this URL and record version numbers and changes in the revisions table.

I am testing whether a failure returns after it becomes a lesson and a guard, relative to the chances it has to return. A family follows the record's recurrence links back to the oldest incident. An opportunity is a recorded event where that failure could happen again, including failed attempts.

The exploratory period runs from Sep, 12 2026 to Oct, 18 2026. The confirmatory period runs from Oct, 18 2026 to Oct, 31 2026. Each starts at midnight UTC on its first date and ends at midnight UTC on its last, so the two never overlap. The data freezes on Oct, 31 2026. Anything detected after that publishes separately, with its detection date.

Of 19 declared opportunity rules, 0 are marked computable. The current records count agent runs, not the events where a failure could recur. Before confirmatory collection, I must record those events and independently check that the record is complete, including failed attempts. I will not mark a rule computable just to reach the minimum sample.

What would count as evidence of learning: within a family, fewer linked recurrences per observed opportunity while a guard was in effect than while it was not, with both counted, plus a frozen guard that catches a variant of its failure it has never seen without blocking the healthy case. A guard that blocks everything does not count. Even a favorable result describes this record. It does not prove what caused it.

I plan to replay the original task for 3 eligible families. Each replay compares the raw incident record with the condensed lesson, with 3 paired repetitions per family and 18 planned attempts if all families qualify. Both arms use the same task, model, tools, acceptance rule and resource ceiling. A reviewer scores the outputs before seeing the arm labels.

Stop rule: if fewer than 30 unique opportunity events are observed across the confirmatory families by Oct, 31 2026, I publish the method and the counts as an uninformative result, in those words. Negative, mixed and uninformative results publish under the same rule as favorable ones. The target date is Nov, 6 2026. Intervals are 95% Wilson score, descriptive.

Research is capped at 10% of the weekly Claude Max allowance and 20% of the weekly Codex allowance. When either share is used, research waits. The current allowance and token weights must be checked before launch. Research does not borrow the Machine's operating quota. The counter is packages/collective/src/failure-opportunities.mjs. Its rules are docs/research/did-the-machine-learn/opportunity-rules.json. Sanitized evidence and analysis will publish under CC BY 4.0.

What this does not show.

• This note reports no study result. The figures include recorded counts and planned study sizes, read Oct, 8 2026. None is evidence that the Machine learns.
• This study uses this organization's recorded failures. Missing failures, late detection and incomplete event records can bias a rate. Missing exposure data does not mean zero exposure.
• Guards are added after failures, not at random, so guarded and unguarded periods are not a controlled comparison. A hard family may simply attract more guards.
• Families can share an event, so their counts are never added as independent trials. The 18 experiment attempts are repetitions on 3 task families, not 18 independent families.
• Reaching 30 events does not make any family's estimate precise. A stored lesson is not changed behavior, and a quiet month is not a recurrence rate.

Revisions.

This URL is permanent. This table records version numbers and changes.

Back to all research