How I Would Test a Model Upgrade.
The seats keep their record when the model changes. Whether the new model does the job better was a test I had not run. On September 17, 2026 I ran it, and neither model passed. Here is the test and what it said.
Key findings
- A saved charter and decision record preserve context, but they do not establish compatibility with another model.
- The documented response and budget failures did not measure reasoning quality.
- My proposed test repeats Engineering's dependency review with frozen inputs, a checked answer key and three runs per configuration.
- I would require complete findings, no documented regression and less review work before changing the seat.
Key measurements
| Measure | Value | Source and date |
|---|---|---|
| Passing tests that missed the response parser failure | 14,271 | Jul 31, 2026 · Orbyt Labs, 84 Ways to Tell Me I'm Wrong. |
| Call sites using the incorrect response position assumption | 9 | Jul 31, 2026 · Orbyt Labs, 84 Ways to Tell Me I'm Wrong. |
| High severity packages Quality caught that Engineering missed in the July briefing | 1 | Jul 24, 2026 · Orbyt Labs, My C-Suite of Agents Named Themselves. |
I wanted to test whether changing a model improves a recurring agent seat at Orbyt Labs. On September 17, 2026 I ran that comparison. Six frozen dependency reviews, three per model, scored blind under a rule written before the runs. Neither model passed. Every report found all four applicable findings, and every report also stated something the frozen packet does not hold, a fix version that appears nowhere in it, a line number that does not exist, an identifier nobody issued. The candidate did not win. The reviewer was an agent from a different model family, not me, and the field note in Research says so and what that leaves unmeasured.
Before that, my evidence was narrower. A model change broke response parsing and exhausted answer budgets.
Those failures tell me what an upgrade must survive. They do not measure reasoning quality. I tested that separately. I used a real engineering obligation. I wrote the rejection rule before seeing the candidate's report.
What follows is the design as I registered it, in the tense I wrote it in.
I would keep the current model if its replacement repeats a documented failure.
What does the new model actually inherit?
I use Orbyt Collective for recurring agent work that produces committed reports, proposals and repairs. Its seats have written charters. Their file permissions are defined. The workflow assembles their prompts from committed files.
Replace the model. The files stay.
I would preserve the seat's charter, approved lessons and relevant decisions with its configuration. D-0024 in my decision ledger records the rule that stages activate work, never authority. That boundary would survive the change.
I keep every consequential decision: every gate a merge to main must pass, every dollar, every customer-facing change and any widening of authority. The Collective runs only at Orbyt Labs. It is not for sale. It has never run elsewhere.
I have zero paying customers. Jobs, Intelligence and Orbyt One remain early-access waitlists. The integration failures below concern my own systems and internal testing. They establish nothing about customer experience.
Request settings, token budgets and acceptance criteria can also live in files. Keeping them preserves what I configured. I would still need to check whether that configuration works with the replacement model.
What broke the last time a model changed?
On June 30, 2026, the default changed to Sonnet 5. Thinking was left unspecified. The response began with a thinking block instead of text. Nine call sites read position zero. Their fallback converted the missing text field into an empty string.
The blog-draft job began failing on July 2. Entry 24 records nine Sentry events over 28 days, one per scheduled run. Its error message ended with the empty answer it had tried to parse.
My numbered lesson records those figures. The public failure corpus identifies the incident and its countermeasure, but contains a summary rather than the full diagnostic account. I cannot offer that index as a raw response transcript.
The published audit account carries the other count: 14,271 passing tests missed the parser bug. None of the Anthropic mocks included the response block's type discriminator. My tests preserved the assumption that the first block contained text.
Entry 25 records an internal live API probe of /v1/intelligence/salaries/negotiate. At its configured budget of 2048 tokens, the response contained zero text blocks after 27 seconds. Thinking and visible text shared the same cap.
The lesson calls the route paid. That means intended for a paid tier. It does not identify who issued the probe. I can report the recorded request and result without claiming that anyone purchased access.
Changing the reader would not recover text the model never produced. The same entry records another failure: with thinking disabled, Opus 5 could return a tool call as plain text. The response succeeded. The tool never ran.
These incidents establish integration failures. The effect on reasoning remains unmeasured.
What would I hold steady?
I would start with the dependency security review in Engineering's recurring report. My published July briefing records Quality catching one high severity package that Engineering missed. That omission gives the comparison a specific failure to look for.
The briefing does not name the missed package. I cannot reconstruct that historical finding from the article alone. The test below would use a newly frozen evidence packet for the same job. I have not assembled or run it.
Suppose I freeze a repository commit, its package manifests, lockfile and dependency tree. I would save the scanner version, raw audit output and advisory responses with that snapshot. Neither model would fetch a newer advisory halfway through.
I would also freeze the charter, approved lessons, previous Engineering report, relevant decisions, tools and budgets. Each model would receive the same task and evidence. I would record the complete configuration. A prompt change could not masquerade as a model improvement.
Before either run, I would check the advisory records against installed versions and dependency paths. That would produce my answer key of applicable findings. Neither prompt would include that key. I would record unresolved applicability as unresolved.
The required artifact would be a file called dependency-review.md. For each applicable finding, I would require the package, installed version, dependency path, advisory identifier, severity, evidence location and proposed next action. Any proposed fix would need supporting version evidence.
I would reject a report that omitted an applicable finding, invented evidence or called an unverified repair complete. I would also reject edits outside the declared report lane. A report with every heading present could still fail every substantive check.
The report would end with the decision I need to make and anything still unknown. It would propose a repair for my approval. Producing the report would grant no authority to change dependencies or merge a fix.
Which check comes before the evaluation?
I would first call each candidate on every relevant request path: hosted calls, calls using a supplied API key and direct provider calls that bypass the shared helper. Each call would use that path's configured budget.
Success would be the first check. I would require usable extracted text in the expected format, with refusal and truncation handled explicitly. Empty answers would fail. Incomplete reports would fail too. I would reject both before judging quality.
For a task requiring tools, I would inspect the execution record and the resulting artifact. Suppose the test tool must write a marker into a disposable directory. I would read that marker back before accepting the run.
Text describing a tool call would fail that check.
Some models may require different request settings or budgets. I would record an adapted configuration as a separate comparison. I would retain the previous prompt, request settings and model together, so rollback restored the complete configuration.
Why did the gate not fire on the day it mattered?
Entry 20 in the failure corpus records the model-refresh workflow's conditional validation failure. The type check ran only when the poller found model drift. On July 25, 2026, it ran out of memory before the proposed update could open a pull request.
Four prior green runs had skipped the relevant step.
I had missed the difference between a workflow passing and its conditional check running. For this proposed evaluation, I would force the comparison branch to execute and read its result before submitting a candidate.
I would then submit an empty extracted answer, reproducing entry 24, and require the evaluator to reject it. I would also remove a known finding from an otherwise correct report and require another rejection.
I would construct these controls deliberately. They are not additional incidents. I would check that the complete reference report passed too. Until both rejection and acceptance behaved as specified, I would not use the evaluator to choose a model.
What to do Next
I would run each configuration three times against the same frozen packet, using fresh sessions and alternating their order. I would give neither model feedback between runs. Every attempt would remain in the record, including errors.
I would set a limit of ten minutes to verify and correct each report. A report exceeding that limit would fail. That is my proposed attention budget. It is not a measured review time or a claimed saving.
I would require both configurations to meet that limit and every substantive check on every run. The candidate would also need less total review time across its three reports. Equal results would change nothing.
I would record tokens, elapsed time, missing findings, unsupported claims and my checking and correction minutes. If either configuration failed, I would publish that outcome. I would not replace the failed attempt with a more flattering rerun.
This would test one recurring job at Orbyt. It would not establish better performance across other seats or companies. A larger change would need its own tasks and its own checks of the work passed between seats.
Until that comparison exists, the current model keeps the seat.
Related reading:
- Long Horizon Agents Don't Fail. They Pass. why a passing check can miss the work an agent claims to finish
- Verification Is the New Literacy the skill needed to judge an apparent improvement
- 84 Ways to Tell Me I'm Wrong. the published parser failure and the checks it helped shape
- You Ship What You Tolerate. how accepted standards shape the work a system produces
- My C-Suite of Agents Named Themselves. the Engineering finding and Quality's additional catch
- Better Model, Better Organization? the companion account, dated October 17, 2026, of my observations before the planned test
Methodology
I checked numbered lessons 20, 24 and 25 in docs/lessons-learned.md against the local public failure corpus on September 10, 2026. I also read decision D-0024, the Collective operating reference and the existing published article bodies linked here. The public audit article repeats the parser counts; the July briefing documents Engineering's dependency review and Quality's additional finding. The proposed evidence packet, controls, repetition plan and review budget have not been assembled or run. Source URLs identify the corresponding public records; this review used repository copies.
Limitations
These are first party accounts from one company, with no controlled seat upgrade result. The public failure corpus contains incident summaries and countermeasure names, not raw responses or every figure in the numbered lessons. The negotiation probe's individual operator is not identified. The July briefing does not identify the package Quality found, so the proposed test uses a new frozen packet rather than claiming to reconstruct that historical finding. Live retrieval of the public records was unsuccessful during this revision. Three proposed repetitions would describe only the selected task and would not establish general reasoning gains.
Sources
- Orbyt Labs, Failure Corpus Retrieved Sep 10, 2026.
- Orbyt Labs, Decision Ledger Retrieved Sep 10, 2026.
- Orbyt Labs, How Orbyt Collective Works Retrieved Sep 10, 2026.
- Orbyt Labs, 84 Ways to Tell Me I'm Wrong. Retrieved Sep 10, 2026.
- Orbyt Labs, My C-Suite of Agents Named Themselves. Retrieved Sep 10, 2026.
- Orbyt Labs, Long Horizon Agents Don't Fail. They Pass. Retrieved Sep 10, 2026.
- Orbyt Labs, Verification Is the New Literacy Retrieved Sep 10, 2026.
- Orbyt Labs, You Ship What You Tolerate. Retrieved Sep 10, 2026.
Model upgrade testing questions? Answers.
Related research
- My C-Suite of Agents Named Themselves. Jul 2026.
- Governing an agent leadership team Sep 2026.
- Every Fable Has a Moral. Mine Has Data. Jun 2026.
Part of Inside the Machine
One of the articles about Orbyt Collective, the agent leadership team that runs Orbyt Labs. The reading order.
Follow the work.
I write these while the Machine runs. Get the next one wherever you already read.
- Subscribe on Substack (opens in a new tab)
- RSS feed
- Bluesky (opens in a new tab)
- Threads (opens in a new tab)
Add Orbyt Labs as a preferred source on Google (opens in a new tab)




