A Model Upgrade on a Frozen Dependency Review

The data.

How it was measured.

The Astra follow-up used the original packet, task prompt, answer key and substantive acceptance rule. Each attempt began in a fresh directory, and file hashes checked the frozen inputs and permitted report lane.

Codex requested gpt-6-astra with high reasoning effort. Its arguments requested disabled network access, web search, apps and subagents. Provider identity in its CLI events: not reported; requested configuration recorded.

The substantive reviewer was a fresh claude-sonnet-5 agent session. It received shuffled report letters, the frozen evidence and the acceptance rule in a separate reading directory. Verdicts were locked before the letter map was opened.

The reviewer recorded 0 missing applicable findings and 0 unsupported claims. Successful executions with a report and no write-lane violation: 3 of 3. Files changed outside that lane: 0.

On Sep, 17 2026, the original comparison's reviewer accepted 0 of 3 claude-sonnet-4-5-20250929 reports and 0 of 3 claude-sonnet-5 reports. Its recorded candidate win remains no; those reports were not rerun or rescored for this follow-up.

What this does not show.

• One task at one organization, with 3 Astra attempts. The historical Sonnet runs are context, not a contemporaneous control. This does not establish that one model is better across other tasks or companies.
• Codex and Claude use different built-in instructions and tool implementations. This follow-up used Codex shell and patch tools, high reasoning effort and a process timeout; the original comparison used Claude file tools and a turn limit. A difference cannot be attributed to the model alone.
• The reviewer also changed: an OpenAI agent reviewed the original Claude reports, while a Claude agent reviewed the Astra reports. Differences between the recorded verdicts can reflect the reviewer as well as the report author.
• The founder did not perform the substantive reads. Human review minutes and the original attention-saving requirement were not measured. Agent acceptance does not authorize replacing the seat's model.
• The unchanged mechanical detector has known formatting false positives. Its rejection count is reported separately from the agent's substantive verdicts. An agent reviewer can also make mistakes.
• After the review was locked, the operator found that 3 reports described a failed subagent launch whose call and error were absent from the saved CLI events. The reviewer did not flag those execution claims. They remain unverified; their absence from this event format does not prove fabrication. The locked scores are retained, and the runs are not presented as fully verified autonomous operation.
• Token totals are the Codex CLI's reported input plus output for report attempts only, with cached input counted once. They exclude setup and review and are not a cross-provider cost or quota comparison.
• The temporary workspaces restrict writes and network access; they are not filesystem jails. The task prohibits outside reads, and the recorded tool activity must be checked separately from file hashes.
• The first compatibility control failed before reaching the model because Codex could not initialize inside the outer sandbox. That attempt is retained. A permitted infrastructure retry passed the control before report generation began.
• A second comparison block, the current model against claude-opus-5-5 on the same frozen packet under the same rule, is pre registered and partly run, and none of its reports has been read, so it has no result yet.

Revisions.

This URL is permanent. This table records version numbers and changes.

Back to all research