Safety Is a Default, Not a Debate.

You cannot make an unpredictable system safe. I wrote about why that is formally true and why I build on it anyway.

But there is a decision fully inside your control, and it is the entire subject of this post: what happens when it fails.

That decision is not a debate, a policy, or a review meeting. It is a default you ship. At 2am, when the check breaks and nobody is watching, the system consults exactly one thing: the failure behavior you wrote down months earlier.

Safety is a default, not a debate. Everything dangerous in my stack fails closed.

What does fail-closed mean, concretely?

When a check cannot complete, the system denies instead of allows.

On Orbyt, a subscription gate that breaks refuses access rather than granting it. A cron route that cannot find its secret refuses the request, through a null check _before_ the comparison, because a missing secret compared against a missing header is a match. An unverifiable state stops the pipeline rather than shipping on hope.

The fail-safe (opens in a new tab) principle is a century old in heavy machinery: the elevator brake engages when the cable slacks, not when a sensor confirms disaster. Software forgot it, because in software the failure mode of open feels like uptime. Every outage becomes an open door, and the door stays open exactly as long as nobody notices.

Fail-closed turns the outage into a locked door instead. Locked is recoverable. Leaked is not.

Does your kill switch outrank the work?

Every AI operation now claims a kill switch. Almost nobody asks the follow-up: does the switch _outrank_ the actions it is supposed to stop?

Mine is checked before any push, by an ordering law a mechanical guard enforces: no push step may precede the halt check in any workflow, and the build fails if one does. Because a kill switch consulted after the work ships is not a control. It is a diary entry. That ordering law is 1 of 103 mechanical guards that run before anything of mine merges, and 35 of my 58 recorded failures are now guards like it.

I know the ordering matters because an outside reviewer found the gap: a rebase could import the switch file without stopping the subsequent push. The fix I shipped that afternoon was itself defective, corrected seven minutes later by a commit whose subject line says so. Safety engineering is not a state of grace. It is a loop that keeps finding its own holes.

Blast radius is a union, not a list

The second discipline: measure what a failure can reach by combining permissions, never by reading them one at a time.

Reading secrets is fine. Outbound network is fine. Push access to the deploying branch is fine. Together they are an exfiltration path, and I only saw it when I computed the union on my own agent fleet: 391 individually reasonable allows that summed to an unreasonable capability.

The structural answers are boring on purpose. Credentials scoped to the single step that needs them, so a general-purpose step cannot spend money. 0 of 12 seats can merge to the deploying branch. Lanes drawn by consequence, so the question is never "is this command safe" but "what does this path reach."

What does failing closed cost?

Honest ledger, both columns.

The cost: my gates discarded 30 of the last 51 agent runs. Real work, real tokens, thrown away because something drifted out of lane or failed a check on the way out. Fail-closed means paying for false alarms.

The payout: in June a safeguard failure at the model layer took my best model away for 19 days, by government order. The incident cost a one-day model swap instead of a shutdown, because nothing the company runs on lived in the model. The trust lived in the system around it: the tests, the docs, the harness, the fail-closed gates. The safety P&L is not hypothetical for me. It has already settled once.

That is the trade in one line: a misfiring gate costs a retry. A missing one costs an incident. Slow is what unsafe looks like, after.

What to do Next

List every surface where your AI can spend money, grant access, publish something, or delete something. Four columns, rarely more than a dozen rows.

For each row, answer one question: _when the check on this path breaks, does the system allow or deny?_

Every "allow" on that list is a policy decision you made by not making it. Convert them. Most are one conditional and a test, and each one moves a future incident into the retry column.

Then find your kill switch and ask what it outranks. If the honest answer is "nothing, structurally," you have a hope with a name.

The default you ship is the only safety policy the failure will ever read.

Related reading:

• Why AI Safety Matters the formal case this post is the engineering consequence of
• Obedient Is Not Aligned. the intent gap the defaults exist to contain
• 391 Yeses and Not One No. blast radius as a union, measured on my own fleet
• Human in the Loop Is Not Enough why the human needs structure, not just presence