A Model Upgrade on a Frozen Dependency Review

The data.

How it was measured.

Locked reviewer verdicts and mechanical verdicts come from the unblinded result. Output tokens come from each run receipt.

The reviewer was the agent. The review time clause was not measured: an agent read the reports (Deviations 4). Equal review totals leave the model unchanged: no.

The registered rule requires every run in each arm to pass and the candidate to use less total review time. Substantive acceptance alone does not establish a win.

The reviewer found 12 unsupported claims across the current model reports and 14 across the candidate reports. An unsupported claim is a sentence the frozen packet does not back.

What this does not show.

• A single seat, a single task and 3 runs per arm. This does not establish performance across other seats or companies.
• The founder did not perform the substantive reads. The reviewer was an agent of a different model family from the reports' authors, reading in a copy of the repository with the letter map and the named run directories removed, and with no network. No human minutes were recorded, so the review time comparison in the pre registered rule was not made.
• The mechanical label detector was calibrated on a single rendering. It rejected 3 current model reports and 3 candidate reports. The protocol records the calibration failures and a second calibration proven after unblind. The evaluator stayed unchanged for this trial.

Revisions.

This URL is permanent. When the data is refreshed the version bumps and a row lands here, so a citation made today still resolves to the finding it cited.

Back to all research