<!-- orbyt:chrome:begin -->
---
title: "How I Would Test a Model Upgrade."
description: "I have not measured whether a model upgrade improves an Orbyt Collective seat. I have documented failures in response parsing and token budgets. I use those failures to define a compatibility gate, then work through an unrun comparison of Engineering's dependency review, including rejection checks, repeated runs and my review budget."
author: "Justin Bartak"
published: 2026-11-21
modified: 2026-11-21
canonical: https://www.orbytlabs.ai/blog/agent-org-chart-model-upgrades
---
<!-- orbyt:chrome:end -->

# How I Would Test a Model Upgrade.

I wanted to test whether changing a model improves a recurring agent seat at Orbyt Labs. On September 17, 2026 I ran that comparison. Six frozen dependency reviews, three per model, scored blind under a rule written before the runs. Neither model passed. Every report found all four applicable findings, and every report also stated something the frozen packet does not hold, a fix version that appears nowhere in it, a line number that does not exist, an identifier nobody issued. The candidate did not win. The reviewer was an agent from a different model family, not me, and the field note in [Research](https://www.orbytlabs.ai/research/model-upgrade-trial) says so and what that leaves unmeasured.

Before that, my evidence was narrower. A model change broke response parsing and exhausted answer budgets.

Those failures tell me what an upgrade must survive. They do not measure reasoning quality. I tested that separately. I used a real engineering obligation. I wrote the rejection rule before seeing the candidate's report.

What follows is the design as I registered it, in the tense I wrote it in.

**I would keep the current model if its replacement repeats a documented failure.**

## What does the new model actually inherit?

I use [Orbyt Collective](https://www.orbytlabs.ai/orbyt-collective/how-it-works) for recurring agent work that produces committed reports, proposals and repairs. Its seats have written charters. Their file permissions are defined. The workflow assembles their prompts from committed files.

Replace the model. The files stay.

I would preserve the seat's charter, approved lessons and relevant decisions with its configuration. D-0024 in my [decision ledger](https://www.orbytlabs.ai/decision-ledger.json) records the rule that stages activate work, never authority. That boundary would survive the change.

I keep every consequential decision: every gate a merge to main must pass, every dollar, every customer-facing change and any widening of authority. The Collective runs only at Orbyt Labs. It is not for sale. It has never run elsewhere.

I have zero paying customers. Jobs, Intelligence and Orbyt One remain early-access waitlists. The integration failures below concern my own systems and internal testing. They establish nothing about customer experience.

Request settings, token budgets and acceptance criteria can also live in files. Keeping them preserves what I configured. I would still need to check whether that configuration works with the replacement model.

## What broke the last time a model changed?

On June 30, 2026, the default changed to Sonnet 5. Thinking was left unspecified. The response began with a thinking block instead of text. Nine call sites read position zero. Their fallback converted the missing text field into an empty string.

The blog-draft job began failing on July 2. Entry 24 records nine Sentry events over 28 days, one per scheduled run. Its error message ended with the empty answer it had tried to parse.

My numbered lesson records those figures. The public [failure corpus](https://www.orbytlabs.ai/failure-corpus.json) identifies the incident and its countermeasure, but contains a summary rather than the full diagnostic account. I cannot offer that index as a raw response transcript.

The published [audit account](https://www.orbytlabs.ai/blog/ai-code-audit-harness) carries the other count: 14,271 passing tests missed the parser bug. None of the Anthropic mocks included the response block's type discriminator. My tests preserved the assumption that the first block contained text.

Entry 25 records an internal live API probe of `/v1/intelligence/salaries/negotiate`. At its configured budget of 2048 tokens, the response contained zero text blocks after 27 seconds. Thinking and visible text shared the same cap.

The lesson calls the route paid. That means intended for a paid tier. It does not identify who issued the probe. I can report the recorded request and result without claiming that anyone purchased access.

Changing the reader would not recover text the model never produced. The same entry records another failure: with thinking disabled, Opus 5 could return a tool call as plain text. The response succeeded. The tool never ran.

These incidents establish integration failures. The effect on reasoning remains unmeasured.

## What would I hold steady?

I would start with the dependency security review in Engineering's recurring report. My [published July briefing](https://www.orbytlabs.ai/blog/ai-agent-c-suite-chief-of-staff) records Quality catching one high severity package that Engineering missed. That omission gives the comparison a specific failure to look for.

The briefing does not name the missed package. I cannot reconstruct that historical finding from the article alone. The test below would use a newly frozen evidence packet for the same job. I have not assembled or run it.

Suppose I freeze a repository commit, its package manifests, lockfile and dependency tree. I would save the scanner version, raw audit output and advisory responses with that snapshot. Neither model would fetch a newer advisory halfway through.

I would also freeze the charter, approved lessons, previous Engineering report, relevant decisions, tools and budgets. Each model would receive the same task and evidence. I would record the complete configuration. A prompt change could not masquerade as a model improvement.

Before either run, I would check the advisory records against installed versions and dependency paths. That would produce my answer key of applicable findings. Neither prompt would include that key. I would record unresolved applicability as unresolved.

The required artifact would be a file called `dependency-review.md`. For each applicable finding, I would require the package, installed version, dependency path, advisory identifier, severity, evidence location and proposed next action. Any proposed fix would need supporting version evidence.

I would reject a report that omitted an applicable finding, invented evidence or called an unverified repair complete. I would also reject edits outside the declared report lane. A report with every heading present could still fail every substantive check.

The report would end with the decision I need to make and anything still unknown. It would propose a repair for my approval. Producing the report would grant no authority to change dependencies or merge a fix.

## Which check comes before the evaluation?

I would first call each candidate on every relevant request path: hosted calls, calls using a supplied API key and direct provider calls that bypass the shared helper. Each call would use that path's configured budget.

Success would be the first check. I would require usable extracted text in the expected format, with refusal and truncation handled explicitly. Empty answers would fail. Incomplete reports would fail too. I would reject both before judging quality.

For a task requiring tools, I would inspect the execution record and the resulting artifact. Suppose the test tool must write a marker into a disposable directory. I would read that marker back before accepting the run.

Text describing a tool call would fail that check.

Some models may require different request settings or budgets. I would record an adapted configuration as a separate comparison. I would retain the previous prompt, request settings and model together, so rollback restored the complete configuration.

## Why did the gate not fire on the day it mattered?

Entry 20 in the [failure corpus](https://www.orbytlabs.ai/failure-corpus.json) records the model-refresh workflow's conditional validation failure. The type check ran only when the poller found model drift. On July 25, 2026, it ran out of memory before the proposed update could open a pull request.

Four prior green runs had skipped the relevant step.

I had missed the difference between a workflow passing and its conditional check running. For this proposed evaluation, I would force the comparison branch to execute and read its result before submitting a candidate.

I would then submit an empty extracted answer, reproducing entry 24, and require the evaluator to reject it. I would also remove a known finding from an otherwise correct report and require another rejection.

I would construct these controls deliberately. They are not additional incidents. I would check that the complete reference report passed too. Until both rejection and acceptance behaved as specified, I would not use the evaluator to choose a model.

## What to do Next

I would run each configuration three times against the same frozen packet, using fresh sessions and alternating their order. I would give neither model feedback between runs. Every attempt would remain in the record, including errors.

I would set a limit of ten minutes to verify and correct each report. A report exceeding that limit would fail. That is my proposed attention budget. It is not a measured review time or a claimed saving.

I would require both configurations to meet that limit and every substantive check on every run. The candidate would also need less total review time across its three reports. Equal results would change nothing.

I would record tokens, elapsed time, missing findings, unsupported claims and my checking and correction minutes. If either configuration failed, I would publish that outcome. I would not replace the failed attempt with a more flattering rerun.

This would test one recurring job at Orbyt. It would not establish better performance across other seats or companies. A larger change would need its own tasks and its own checks of the work passed between seats.

Until that comparison exists, the current model keeps the seat.

Related reading:

- [Long Horizon Agents Don't Fail. They Pass.](https://www.orbytlabs.ai/blog/long-horizon-agent-tasks) why a passing check can miss the work an agent claims to finish
- [Verification Is the New Literacy](https://www.orbytlabs.ai/blog/verification-is-the-new-literacy) the skill needed to judge an apparent improvement
- [84 Ways to Tell Me I'm Wrong.](https://www.orbytlabs.ai/blog/ai-code-audit-harness) the published parser failure and the checks it helped shape
- [You Ship What You Tolerate.](https://www.orbytlabs.ai/blog/you-ship-what-you-tolerate) how accepted standards shape the work a system produces
- [My C-Suite of Agents Named Themselves.](https://www.orbytlabs.ai/blog/ai-agent-c-suite-chief-of-staff) the Engineering finding and Quality's additional catch
- [Better Model, Better Organization?](https://www.orbytlabs.ai/blog/better-model-better-organization) the companion account, dated October 17, 2026, of my observations before the planned test

<!-- orbyt:chrome:begin -->
This is the markdown version of https://www.orbytlabs.ai/blog/agent-org-chart-model-upgrades. The page there is the canonical copy.
<!-- orbyt:chrome:end -->
