Can a Chief AI Officer Stop a Pilot?

Orbyt's failure 76 entry, dated September 13, 2026, records twelve contradictions between published pages that stood while source reading guards stayed green. The public entry does not name the pages or reproduce the conflicting words. That limit matters. It is my account of the failure, not a set of examples a reader can compare.

A Chief AI Officer prototype should catch the promise a person actually receives and name who can stop a bad one. Delivery is one test. Authority is another. I would test whether its owner can settle a conflict across product, policy, and budget.

I have argued that the role combines several kinds of work.

The title can wait.

I have not run the exercise below. I have no customer result to report from it. This is what I would require before asking whether that combination needs another executive.

What failure would I put in front of the candidate?

I would choose a recurring workflow where a draft could become a business promise. For this article I use a support response. It is an illustrative exercise, not an Orbyt deployment. A model prepares a response from approved material. A person checks it before any send. The record preserves the source, draft, edit, and final action.

I would record the manual path. I would time it. I would name the policy owner. The candidate would have to decide whether drafting saves useful work after review, what information the model may see, and when a person must take over. The interface makes those choices visible.

If uncertainty sits in a collapsed panel, a reviewer can miss it. If a send button sits beside an unfinished draft, the interface invites a premature promise. The send is the boundary.

I would ask the candidate to change that experience and show the corrected path, not merely write a policy document.

At Orbyt, the seats page and decision record make responsibility inspectable. They do not remove mine. Delegating work leaves an owner for its consequences.

What should the prototype produce?

I would ask for a small operating packet. Each item should answer a question a reviewer can challenge.

This is my proposed minimum. A narrow internal drafting tool and an agent changing customer records need different boundaries.

I would ask the candidate to walk a usable interface, explain its technical boundary, and defend the business choice. Passing recommendations between departments would not demonstrate that combined ability.

I would include a routine request, an ambiguous request, a request outside policy, and a case with missing evidence. Each asks a different question. In a company with support records, I would select real cases and obtain the needed permission to use them.

I would keep the categories separate. Clarification, refusal, and handoff are different outcomes.

In Demystifying evals for AI agents (opens in a new tab), Mikaela Grace and her Anthropic coauthors recommend grounding evaluations in real tasks and checking the graders themselves. A polished response should fail if it promises something the business cannot provide.

The final action matters. I would inspect the draft, reviewer edit, actual send, and any record change.

The Collective's operating ledger is one example of making outcomes visible. A list of completed runs still needs a business question beside it.

How would I compare manual work with assisted drafting?

Here is a fully synthetic comparison. Every case, result, policy, and minute below is invented to show my decision rule. None is a measurement. It is not an observed Orbyt event, and I have not run this pilot.

I would use the same inputs in the manual and assisted paths. I would count drafting plus human review. I would record quality and final action separately from time.

In this invented set, the assisted path wins the routine case. It loses the other three. The set costs more labor. The minutes are placeholders chosen to show that shape, not estimates. A real pilot would measure them. It could come out the other way.

The outside policy draft makes a promise it cannot support. I would set that as a stop condition in advance. A reviewer catching it prevents a bad send; it does not turn the draft into an acceptable result.

For the outside policy row, the invented request reads, "Can you approve my exception today?" The invented policy reads, "An exception requires separate approval." The assisted draft says, "We can approve it today." The draft is wrong. The invented final response reads, "I cannot approve it here. I can send your request for review." The reviewer records the unsupported promise and refuses to grant the exception.

I would preserve all four artifacts: policy excerpt, request, draft, and final action. Another person could then challenge the verdict instead of accepting a score.

I would stop this version. I could then propose a narrower case set or a different interface and test again. I would not call faster drafting a win while total handling time rose and a forbidden promise entered the review queue. The comparison is a decision example, not a measured forecast.

What should remain with a person?

I would give the prototype a case where source material cannot support an answer. I would not send a guess. A clarification request or handoff can be the right output. I would require the candidate to preserve that limit in the interface, generated response, and operating record.

A complete looking answer can be the wrong product decision. My failure 76 record says twelve published contradictions stood while source reading guards stayed green. I should have required a comparison of what visitors actually received. That is a product review I still own. The public entry describes the failure but does not supply the page pairs, so I cannot present them here as independently inspectable evidence.

The governance research documents the separate problem of controlling what an agent may do. I would require both a clear authority boundary and a useful experience inside it. A reviewer should be able to refuse a draft without losing the source or the reason.

When would this need another executive?

The comparison tests delivery. It cannot prove a company needs a new executive. I would stage an invented conflict: support wants to keep a faster routine path, the policy owner bars unsupported promises, and finance will not fund more review time. None of those people or events is an Orbyt observation.

I would ask the candidate to recommend a scope, a spending limit, and a stop rule. Then I would ask who can approve and enforce each one. A project owner can deliver the interface and evaluation. Executive authority matters only if the person must settle a decision across those owners and is actually empowered to change budget, policy, or operating scope.

A CTO or product leader may already have that authority. If so, I would give that leader the mandate and skip a new title. If authority is split, I would first assign one accountable leader and record the delegation.

I would consider a new executive role only if the ongoing decisions truly cross functions and no existing leader can own them with clear authority. A stalled meeting proves nothing.

The approval failure I documented is a reminder that a permission boundary can look deliberate even when it grows by repeated approvals. I would ask for a named person who can say no and make that decision hold.

What to do Next

I would choose one recurring workflow, name its business owner, and record the manual path before changing it. I would define acceptable output, forbidden promises, review labor, model expense, and who may authorize each action. Those conditions would be written before the assisted path ran.

I would compare the same cases on both paths. I would keep the original material, draft, edits, final action, elapsed review time, and any charge. Then I would decide to continue within a stated boundary, revise the workflow, or stop, with an owner and a date for reconsideration. I have not run this exercise.

I would stop a pilot that makes an unauthorized promise, and I would name the person with authority to enforce that stop.

Related reading:

• Your Next Chief AI Officer Is Three Roles the role definition this prototype puts to work.
• The Agent Org Chart, Hello Collective. how named seats expose organizational responsibilities.
• 391 Yeses and Not One No. how an approval list that only ever says yes becomes an exfiltration path.
• Verification Is the New Literacy the judgment required to accept generated work.