AI slop is work that looks finished and is not: it compiles, it reads well, and it is wrong. Justin Bartak keeps it out of his software by testing every change his AI agents make before it ships: a main suite of 39,133 tests on Oct 19, 2026 and, counted October 20, 2026, 46,247 tests across every suite and 100 audit dimensions. When a mistake gets through anyway, it is recorded, and most become a new check. Since July 2026, most new checks have been the agents' own idea.
A team of AI agents, The Machine, runs parts of the company inside the same checks. Everything is connected, so one update spreads everywhere, and the checks are what make that safe.
The best time to catch a mistake is before it exists in the live product. A commit is one saved change to the code. Every commit runs the tests that touch it and the checks that guard the rules, and the full test suite runs before my changes reach the live site. A failed check stops the change where it is. When a mistake does get through, I do not just fix it. I add a check for it, so I make each mistake only once.
Two kinds of mistake are worth naming. Tech debt is the cost of every shortcut you meant to clean up later. It grows quietly until fixing it takes longer than the work itself. AI slop is work that looks finished and is not: it compiles, it reads well, and it is wrong.
A test is a small program that checks one thing the code must do, and fails loudly when it stops doing it. An audit dimension is a whole category of problem the system checks for on its own, like leaked secrets, broken links or a page AI answer engines cannot quote. Lines of code here means every line the sites, apps, agents and console are built from, mostly TypeScript with some Swift and shell, blank lines, comments and article text included, tests not included.
I used to ask for these checks. On Jun 24, 2026, I told the agents they could create new ones as they saw fit, without asking me. The record shows when that took hold: from Jul 11, 2026, most new checks were the agents' idea. I still ask for one now and then.
It works like this. I ask an agent to fix a bug, clean something up or build a page. It does that, then asks what kind of mistake made the work necessary. It writes a check that would have caught it, proves the check fails on the old code and passes on the new, and runs it across the whole codebase. From then on, the check runs with the rest of the audit.
One example. I looked at a pile of loose documents and said, “This is messy I am not sure if we are even using these, are we?” The agent cleaned them up, then added an audit dimension that flags loose documents from then on. I never asked for the check.
Since , 36 of the 40 new audit dimensions started with an agent. Of those, 24 were checks an agent decided to build while doing other work, 6 were checks an agent had already built and then made a dimension, and 6 an agent proposed and I approved before it was built. Across all 100, 38 started with an agent. Each one is tagged in the full list below, with how I classified it.
Step 1Spot a new kind of problem.
Step 2Write a check for it, and prove it can fail.
Step 3Run it across the whole codebase.
Step 4Fix what it finds.
Step 5Keep it running with the rest of the audit.
Eight months, one commit at a time.
Drag across the chart, or use the arrow keys, to watch both lines grow. The test line counts the main test suite, 39,133 tests on Oct 19, 2026; the 46,247 above also counts the browser, iPhone, Collective and console suites. The dimension line counts the audit, and it turns green where the agents took over. Every point is a real commit.
39,133tests in the main suite
100audit dimensions
commit 6cea61614
Tests in the main suite, left axis
Audit dimensions, right axis
Green: audit dimensions after the agents took over and most new checks became their idea,
025k50k050100
MarAprMayJunJulAugSepOct
First tests: 63 of them.First check that runs before every commit.The audit begins with 16 dimensions.I tell the agents they may add checks as they see fit, without asking me.An agent panel chooses five new dimensions.The log that lied gets its check.88 audit dimensions.Sep 23Alignment gauge peaks at 5.6 percent, then recovers.Sep 24100 audit dimensions.My Tests Passed. The Log Was Wrong.
From to , the test count grew from 63 to 39,133 and the audit dimensions grew from 0 to 100. One point per day; the line joins the days.
First tests: 63 of them.
First check that runs before every commit.
The audit begins with 16 dimensions.
I tell the agents they may add checks as they see fit, without asking me.
Three jobs a company would hire for, done by checks.
Each one is a real change. Each check fixed what was already there, and now guards everything that comes after. The receipts are commit ids: each is the fingerprint of one saved change in the code's history.
Staff engineer
Cleanup without a cleanup sprint.
When the same block of code shows up in many places, the agents pull it into one shared piece and move every copy onto it. Copied code drifts. Fix a bug in one copy and the other five still have it.
Before
Six API endpoints, the addresses other programs call for data, each carried about 150 lines of the same setup code, written by hand.
After
One shared piece replaced them. 535 lines came out and behavior stayed identical, down to a known quirk the first lesson below is about, and all 208 tests that check each response’s promised shape stayed green. Five more endpoints moved onto it eleven days later. The house rule is that new read endpoints start from it.
Every check was already passing. A deeper sweep went looking for kinds of risk no check covered yet. A clean result only means you are safe from the problems you already check for.
Before
A sweep across 22 classes of vulnerability, plus 15 hands-on probes of the running app, found nothing critical.
After
Every finding worth fixing was fixed the same day. Then the static half of the sweep became a permanent check that runs in CI on every push. Its first run scored 80 percent: one failure was the new check itself being wrong, the other a reviewed exception. After that it scored 100 percent.
Receipt: Commits 683c6ba04 and ffacda78a (May 13, 2026).
SEO and answer engine specialist
Pages built to be quoted.
AI answer engines quote short, direct answers. A check now reads every content page the way they do. If an engine cannot lift a clean answer from your page, it quotes someone else.
Before
The first run across 79 pages found 41 missing the markup that marks a passage as quotable, and 34 answers too long for an engine to lift.
After
Thirteen answers across eight guides were cut to fit, one from 122 words to 76, and on Aug 30, 2026 the strict check returned zero findings. It still reads every page, and it reports any new answer that runs long.
I have spent more than twenty years building software, and the lesson I paid for most is that the foundation comes first. Retrofitting checks onto a big codebase is slow, expensive and painful, which is why so many teams put it off. So I started early.
The audit started on Mar 21 with 16 dimensions. Orbyt Jobs, Orbyt Intelligence and Orbyt Collective were all built on top of it, and they came together quickly because the checks were already there.
What went wrong, and what each mistake fixed.
It is not perfect. AI agents still make mistakes, including confident wrong answers, and I never claim they do not. Some of those mistakes got through for weeks, and the chains below say so. What I claim is that every mistake found is recorded and is meant to become a check, so the same mistake is caught before it ships again. Right now 73 of the 97 on record have one. Each chain below is a real error.
The log that lied.
What went wrong
One part of the API told the programs calling it that a page did not exist, while its own log wrote down that the request was bad. 13,364 tests passed for a month, because each tested one side and none checked that the two agreed.
How it was caught
Reading the live logs against what the API really returned.
How it was fixed
The API and its log now report the same thing.
The check that prevents it
A new check compares every logged status with the real response. It was proven by putting the bug back and watching it fail.
What it taught the system
It taught the system to test that two things agree with each other.
The month that ran long.
What went wrong
Adding three months to Aug 31 landed on Dec 1 instead of Nov 30, so a subscription pause could end days late. The test did the same wrong math as the code, so the two agreed.
How it was caught
A test run that happened to fall on the last day of a month.
How it was fixed
Adding a month now stops at the last real day of the month.
The check that prevents it
The test checks a range of days and a table of hand-written month-end dates, so it no longer shares the code's assumption.
What it taught the system
Any test that recomputes its own answer the same way the code does is now treated as a red flag.
The answers that came back empty.
What went wrong
An AI model began thinking out loud by default, so every reply arrived with an empty thought in front. Nine places read only that first piece and returned nothing for 28 days.
How it was caught
Tracing why AI features went quiet without an error.
How it was fixed
Every reply is now read by looking for the answer instead of taking the first piece.
The check that prevents it
A check refuses the old pattern anywhere in the code. It was proven by putting the original line back and watching it fail.
What it taught the system
A change in someone else's default is now treated as a change to our code.
The monitor that only looked at 200 rows.
What went wrong
A health check read 200 log rows out of 9,072, found none of the 18 real failures, and reported all clear.
How it was caught
The number it reported was exactly the number of rows it fetched at a time, which is never a real count.
How it was fixed
It now filters for failures first and counts all of them.
The check that prevents it
Its test now runs real log lines through it, and a full batch is reported as cut off, never as clean.
What it taught the system
A monitor that cannot see is now required to say so, instead of saying everything is fine.
The alignment gauge watches whether the agents tell the truth about their own work. It counts how many of their claims a second, independent check contradicted. Its limit is 10 percent.
It peaked on Sep 23, 2026 at 5.6 percent, 1 claim in 18, when an engineering agent reported its checks green and the full test suite refused its work. The work never landed. The cause, a stale file the agent had not written, was found and fixed, and by Oct 16 the gauge read 2.3 percent, 3 claims in 132. This is a snapshot from the founder console, which is private.
Why it matters
The checks are the team: what a thousand-person company means here.
The headline describes the work. I am one person. Companies grow to thousands of people partly because more code needs more people to check it: testers, security engineers, reviewers, accessibility specialists. Here, that checking is written down as code and runs automatically. The full list below maps each audit dimension to the job a company would hire for it.
What is true today is on this page: the counts, the timeline and the receipts. Built foundation first, this kind of system grows by adding checks.
Orbyt Collective, which I call The Machine, is a team of AI agents that each run one department of the company on a schedule: Chief of Staff, Marketing, Product, Quality, Legal, Engineering, Innovation, Security and Finance. Two more, Operations and Revenue, ran earlier and have been wound up; the table below still counts their runs. Each has a written charter and works inside the same checks as the code. See them on Agent Seats.
A human is still in the loop. I decide anything that touches money, public claims or the rules themselves. Full autonomy is the goal, and the Autonomy Ledger publishes how it is going: right now 145 of 162 recorded agent runs, 89.5 percent, made it through the checks to a finished result. The ledger says it plainly itself: a small sample, and the org grades itself.
How the agents divide the work: runs per department, from the Autonomy Ledger.
The checks can tell me whether something is correct. They cannot tell me whether it is right. That part is still mine.
I decide what gets built, in what order, and what it is called. The agents choose how. Every article’s hero artwork is made by hand, and an article with no hero is not published. Agents write the words. A person makes the picture. New screens go through a design review by a panel of agents before they land. It proposes, and one round of its fixes is applied. I do the fit and finish afterward.
A review proposes. It does not remove what I chose. Once one took out a control I had picked, so I put it back and wrote the rule down. The sentences a stranger reads go past a marketing copy review, and my own words stay mine. When the reviewer wanted to soften a line of my creed, the line stayed.
Full autonomy is still the goal, and the Autonomy Ledger shows how far it has got. Taste should be the last thing to go.
Everything is a flywheel.
Nothing I build stands alone. Most pieces read from one source, so one update spreads to every place built from it. That is how one person runs at this scale. It is also why the checks matter most here: because everything is connected, one error could spread everywhere too.
Where one glossary change lands: the commit of Oct 17, 2026 (aeef2b9e5).
The flywheel
Connected systems. One change to a source updates every page, file and feed built from it.
The Machine
Orbyt Collective's agents keep it turning: writing, checking and publishing on a schedule.
The test harness
Checks every commit, so an error is stopped at the source instead of spreading everywhere.
The numbers on this page are the example. One generator runs on every commit and writes the site's stats file. Counts from that one file then land on the About page, the founder page, Building in Public, the Orbyt Collective pages and this page, with nobody typing them. A bigger one: a single glossary commit on Oct 17, 2026 (aeef2b9e5) gave each of 188 terms across four glossaries its own page, and every place those terms had to reach, the sitemap, the search engine notification list, a public dataset, the llms.txt files AI crawlers read and the navigation, is held to the glossary by a test, so none of them can drift.
“A flywheel without guardrails is a liability. With them, it is leverage.”
All 100 of them, in plain words, grouped by the kind of work, with the job a company would staff for it, the day it was added, who started it, and the public record that says so. Quotes are my own words from the build journal, typos left in.
The code builds, the tests pass, and every piece of the product does what it says it does.
Build
Makes sure the whole site still builds from scratch, so a broken change never reaches visitors.
Release engineer · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
TypeScript
Checks that every piece of code agrees on the shape of its data before anything runs.
Software engineer · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
ESLint
Catches risky or sloppy code patterns with an automated reviewer before they ever ship.
Senior software engineer · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
Tests
Runs the full automated test suite, so a change that breaks existing behavior is caught immediately.
QA engineer · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
Coverage
Flags new code that arrives without any tests, so untested logic cannot quietly slip in.
QA engineer · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
Json.parse safety
Makes sure reading saved data never crashes the app when that data is damaged or unexpected.
Software engineer · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
localStorage safety
Makes sure saving data in the browser never crashes the app when storage is full or blocked.
Frontend engineer · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
Dead code
Finds shared helper code that nothing uses anymore, so the codebase stays lean and easy to read.
Software engineer · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
Console.log
Catches leftover debugging messages that would clutter the browser console or reveal internal details.
Frontend engineer · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
Realtime listeners
Checks that screens showing saved data refresh themselves when that data changes on another device or tab.
Frontend engineer · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
Durability
Checks that every feature has regression tests and that offline saving and automatic recovery stay switched on.
QA engineer · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
Public APIs
Checks that the public developer API offers every promised endpoint with the right actions and documentation.
API platform engineer · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
Safari Extension
Checks the Safari extension is packaged correctly, with every file and icon present and no embedded secrets.
Browser extension engineer · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
OpenAPI Contract Validation
Calls the public API and confirms each response matches the shape its published documentation promises.
API platform engineer · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
Envelope Contract
Makes sure every data API response uses one consistent, locked format that developers can rely on for years.
API platform engineer · . Added .
An agent added it on its own.
Evidence: My only prompt that day was to continue; the agent decided on its own to build this regression guard (1ddaf17c6).
Stress Test
Catches the quiet bug types most likely to reach production, like date math that slips a day around midnight.
QA engineer · . Added .
I asked for it.
Evidence: I asked to "add new dimensial test to our /audit strss test projest, pure honesty" (99f1e52c4).
Dual-Mode Stripe Boundary + Outbound Idempotency
Keeps test payments and real payments strictly separated and makes sure no charge can ever be sent twice.
Payments engineer · . Added .
An agent panel chose it.
Evidence: I convened a panel "to see what dimensional test I am missing"; it picked five of thirty five ideas (fd32a116d).
Intra-function tangle + module coupling
Flags functions that are nested too deeply or tangled too tightly to read, test and change safely.
Staff software engineer · . Added .
I asked for it.
Evidence: I asked: "we should create a new audit dimension that makes sure this never happens" (3c2beed68).
Billing state-machine parity
Checks that every product interprets a subscription's status, like paused or canceled, in exactly the same way.
Payments engineer · . Added .
An agent panel chose it.
Evidence: I typed one command, /evolve-audit, and left; the agent panel it convened chose this dimension (6aa9dbbca).
Intelligence contract-surface parity
Makes sure every data API endpoint appears in the docs, the permission list, the developer kit and the AI tools.
Developer experience engineer · . Added .
An agent panel chose it.
Evidence: I typed one command, /evolve-audit, and left; the agent panel it convened chose this dimension (3578f1737).
Anthropic response shape
Catches code that reads an AI provider's reply by position, which silently breaks when the reply format shifts.
AI platform engineer · . Added .
An agent added it on its own.
Evidence: Medium confidence: no request from me was found, and the agent's commit says it mechanized an incident instead of writing a lesson (496528f56).
Orphaned debt
Finds leftover files, text and settings that nothing uses anymore but still mislead anyone searching the code.
Staff software engineer · . Added .
An agent added it on its own.
Evidence: Medium confidence: built inside a hardening loop where I asked not to be questioned, then left unregistered until this commit (eede38b46).
Generated artifact drift
Checks that files made by automated generators match what the generator would produce today, so nothing goes stale.
Release engineer · . Added .
An agent added it on its own.
Evidence: An outside review by Codex raised the finding; the agent chose to answer it with a new dimension (f630912f3).
Customer data stays private, secret keys stay secret, and every door is locked to the people who should not open it.
Security
Scans the whole codebase for leaked secrets and unsafe patterns that could expose customer data.
Application security engineer · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
API auth
Makes sure every private server endpoint checks who is asking before it returns anything.
Application security engineer · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
API rate limiting
Checks that every private endpoint limits how often it can be called, blocking abuse and runaway costs.
Application security engineer · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
Exfiltration Guard
Checks that private data cannot be pulled out of the product through overly open settings or unchecked links.
Application security engineer · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
Security Posture
Verifies that structural protections, like sign in gates and browser security headers, are still in place everywhere.
Security engineer · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
Sast Coverage
Scans the code for dangerous programming patterns that attackers commonly target, before they can ever ship.
Application security engineer · . Added .
I asked for it.
Evidence: I asked: "can we add another dimensional test for this SAST/DAST to our /audit tests" (ffacda78a).
Authorization / Multi-Tenant Isolation
Proves each customer can only ever see their own data, never another customer's.
Security engineer · . Added .
I asked for it.
Evidence: I named the classes a scanner misses and said "build all of these"; the agent picked the final thirteen (8fcef9162).
Software Composition Analysis + license gate
Checks outside code libraries for known security flaws and for licenses that would restrict how we ship.
Application security engineer · . Added .
I asked for it.
Evidence: I named the classes a scanner misses and said "build all of these"; the agent picked the final thirteen (8fcef9162).
Data-lifecycle / right-to-erasure
Makes sure that when a person deletes their account, every piece of their data is actually removed.
Privacy engineer · . Added .
I asked for it.
Evidence: I named the classes a scanner misses and said "build all of these"; the agent picked the final thirteen (8fcef9162).
Supply-chain / build integrity
Checks the build pipeline uses pinned, trusted tools and dependencies, so nobody can slip in tampered code.
DevSecOps engineer · . Added .
I asked for it.
Evidence: I named the classes a scanner misses and said "build all of these"; the agent picked the final thirteen (8fcef9162).
Secret scanning over git history
Searches the project's entire history for passwords or keys that were ever saved, even ones later deleted.
Application security engineer · . Added .
I asked for it.
Evidence: I named the classes a scanner misses and said "build all of these"; the agent picked the final thirteen (8fcef9162).
Dast
Probes a running copy of the site from the outside to confirm sensitive pages and security headers are locked down.
Penetration tester · . Added .
I asked for it.
Evidence: I named the classes a scanner misses and said "build all of these"; the agent picked the final thirteen (8fcef9162).
AI red-team / prompt-injection / MCP abuse
Checks the AI features resist manipulation through crafted inputs and never reveal private data in their answers.
AI security engineer · . Added .
I asked for it.
Evidence: I named the classes a scanner misses and said "build all of these"; the agent picked the final thirteen (8fcef9162).
Apple Privacy Manifest Completeness
Checks the iPhone app declares the device data it uses, which Apple requires before it approves an app.
iOS engineer · . Added .
An agent panel chose it.
Evidence: I convened a panel "to see what dimensional test I am missing"; it picked five of thirty five ideas (fd32a116d).
Org secret exfiltration
Scans everything the AI agents write for passwords or keys before it is saved, and hides any it finds.
Security engineer · . Added .
An agent made an existing check a dimension.
Evidence: An agent turned the org's existing guards into numbered dimensions, as move one of a review I requested (30db0aaf5).
Database privilege posture
Checks that powerful database functions can only be run by the accounts that are meant to run them.
Database security engineer · . Added .
An agent added it on its own.
Evidence: Medium confidence: the agent built this inside a hardening loop where I had asked not to be questioned (033d16d07).
Workflow token scopes
Checks that each automated job asks for exactly the permissions it uses, no more and no fewer.
DevSecOps engineer · . Added .
An agent added it on its own.
Evidence: No request from me was found; the agent added this dimension inside a fix for a broken CI job (d0f78554f).
RLS policy breadth
Checks that database access rules are narrow enough that strangers can never read other people's records.
Database security engineer · . Added .
An agent added it on its own.
Evidence: I told the agent "Do everything." about a findings list; it decided an unowned class needs a dimension (afca2a230).
Migration-sequence RLS final state
Proves that a brand new copy of the database, built from scratch, comes up with every access lock in place.
Database security engineer · . Added .
An agent added it on its own.
Evidence: An outside model's overnight review raised it; I said keep going, and the agent chose a new dimension (2a2659c87).
Supabase Send Email Hook health
Checks that security alert emails, like password changed notices, are actually being delivered to account owners.
Security operations analyst · . Added .
I asked for it.
Evidence: Medium confidence: the agent's commit reports I asked for a detector instead of a rewrite; no quote of mine is public (8c6c2348b).
Agent confinement
Checks that every automated AI agent runs inside a locked sandbox without any stored credentials.
Security engineer · . Added .
An agent made an existing check a dimension.
Evidence: An agent registered this existing safety check after measuring it absent; my prompt was "keep going on this sprint" (f18c4d5e8).
What the database holds, what the code expects, and what a page shows all agree.
markLocalWrite
Checks that every local save is marked for cloud sync, so a person's changes reach their other devices.
Frontend engineer · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
Data Integrity
Checks that the salary data's release labels, dates and prior snapshots all agree with each other.
Data engineer · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
Offline DB-contract drift + rollback safety
Checks that every database table the code relies on is actually defined, and that changes can be safely undone.
Database engineer · . Added .
An agent panel chose it.
Evidence: I typed one command, /evolve-audit, and left; the agent panel it convened chose this dimension (8ec1ab2ae).
Production truth
Compares the live production database against what the code claims, catching problems that tests reading code cannot see.
Data reliability engineer · . Added .
An agent proposed it, I approved.
Evidence: I asked how I can know green tests are real; the agent argued for reading production, and I approved (b3f040152).
RLS policy coverage
Finds locked database tables the app tries to read with the wrong access level, which quietly returns nothing.
Database engineer · . Added .
An agent added it on its own.
Evidence: Medium confidence: built inside a hardening loop where I asked not to be questioned, then left unregistered until this commit (eede38b46).
Migration ledger parity
Checks that every database change applied to production is also recorded as applied, so nothing replays by accident.
Database administrator · . Added .
An agent added it on its own.
Evidence: Medium confidence: no request from me was found; the agent's commit calls the migration ledger a class with no detector (9cb1f9a63).
Shared salary bands
Flags salary pages that show identical figures for different jobs, so no page implies more precision than the data has.
Data analyst · . Added .
An agent added it on its own.
Evidence: I said "Disclose the collision on-page."; the agent added a watcher for the fix nobody is allowed to make (67a741b5f).
Search engines and AI answer engines can find every page, read it, and credit it.
SEO/GEO/AEO
Checks that every marketing page carries the titles, descriptions and structured data that search engines need.
SEO specialist · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
IndexNow sync
Makes sure the list of pages we announce to search engines matches the pages that actually exist.
SEO specialist · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
SEO / AEO / GEO Deep Audit
Audits every indexable page for preferred addresses, social previews and structured data that search and AI engines read.
Technical SEO specialist · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
SEO / AEO / GEO infrastructure durability
Guards the sitemap, redirects, robots rules and AI crawler files that decide what search engines can find.
Technical SEO specialist · . Added .
An agent added it on its own.
Evidence: I asked the agent to "look at the data errror fix things"; it turned what it found into this dimension (20f462c2a).
AEO structure
Checks that articles are structured so AI answer engines can quote them directly and credit the source.
SEO content strategist · . Added .
An agent added it on its own.
Evidence: I said "Go do this you have the authority"; the agent built the detector its own plan had named (daff85f7d).
Every screen works for everyone, on every device, and looks like it came from one team.
iOS App
Checks the iPhone app follows Apple's design conventions and that every source file is part of the build.
iOS engineer · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
Accessibility / Wcag 2.1 AA
Tests key pages against accessibility standards, so people using screen readers or keyboards can use them fully.
Accessibility specialist · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
Color System Hygiene
Makes sure every color comes from the shared palette, so light mode, dark mode and high contrast stay consistent.
Design systems engineer · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
Email mobile-responsiveness durability
Makes sure every email the product sends fits a phone screen without any text getting cut off.
Email developer · . Added .
I asked for it.
Evidence: I asked: "make sure the mobile email issue never happens again, self heal?" (a93f4abd3).
Adversarial Showcase Review
Has independent AI reviewers grade key pages against the best product companies, keeping only findings that survive challenge.
Product design director · . Added .
I asked for it.
Evidence: The LOG records me asking about a thirty sixth dimension, then saying "build it and run it against the launch surfaces" (4a7b586c2).
Pages stay fast for visitors, and no feature can quietly run up the bill.
Cost / metered-resource anomaly guard
Catches page setups that would make the hosting bill balloon, like endless generated pages that crawlers keep fetching.
FinOps engineer · . Added .
I asked for it.
Evidence: I named the classes a scanner misses and said "build all of these"; the agent picked the final thirteen (8fcef9162).
Performance budget
Measures how much code each page makes a visitor download and flags any page that grows too heavy.
Performance engineer · . Added .
I asked for it.
Evidence: I named the classes a scanner misses and said "build all of these"; the agent picked the final thirteen (8fcef9162).
External-Dependency Cost Bound
Makes sure every call to an outside service has a time limit, so one slow vendor cannot run up costs.
FinOps engineer · . Added .
An agent panel chose it.
Evidence: I convened a panel "to see what dimensional test I am missing"; it picked five of thirty five ideas (fd32a116d).
Hosted-AI Token-Accounting Completeness
Makes sure every AI feature records what it spends, so each plan's AI budget is actually enforced.
FinOps engineer · . Added .
An agent panel chose it.
Evidence: I convened a panel "to see what dimensional test I am missing"; it picked five of thirty five ideas (fd32a116d).
Scheduled jobs run, alarms reach a person, and outside services failing never takes the product down with them.
Runtime Smoke
Loads every critical page and endpoint on a running server to confirm each one actually answers.
QA engineer · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
Phase 3 SLO Coverage
Checks that every data API endpoint has a declared speed and uptime target that someone is accountable for.
Site reliability engineer · . Added .
An agent added it on its own.
Evidence: My only prompt that day was to continue; the agent decided on its own to add this coverage guard (1ddaf17c6).
Email / domain auth
Makes sure account emails go through our real mail service from our own domain, so they arrive and are trusted.
Email deliverability specialist · . Added .
I asked for it.
Evidence: I named the classes a scanner misses and said "build all of these"; the agent picked the final thirteen (8fcef9162).
Webhook resilience
Checks that incoming notifications from outside services are verified, handled only once, and never crash on unfamiliar messages.
Backend engineer · . Added .
I asked for it.
Evidence: I named the classes a scanner misses and said "build all of these"; the agent picked the final thirteen (8fcef9162).
Chaos / dependency-failure
Proves that when an outside service goes down, security and spending limits stay shut instead of letting everything through.
Site reliability engineer · . Added .
I asked for it.
Evidence: I named the classes a scanner misses and said "build all of these"; the agent picked the final thirteen (8fcef9162).
Observability / alerting coverage
Checks that errors, failed jobs and cost alarms still reach the founder instead of disappearing without a trace.
Site reliability engineer · . Added .
I asked for it.
Evidence: I named the classes a scanner misses and said "build all of these"; the agent picked the final thirteen (8fcef9162).
Cron Continuity Contract
Checks that every scheduled background job is registered, protected by a secret, tracked, and given a time limit.
Site reliability engineer · . Added .
An agent panel chose it.
Evidence: I convened a panel "to see what dimensional test I am missing"; it picked five of thirty five ideas (fd32a116d).
Model-ID registry parity
Makes sure every AI model the code names is on the watched list, so a retired model gets noticed.
AI platform engineer · . Added .
An agent panel chose it.
Evidence: I typed one command, /evolve-audit, and left; the agent panel it convened chose this dimension (59e9931cf).
Stripe cutover readiness
Checks that switching from test payments to real payments will not quietly break other secrets or settings.
Payments engineer · . Added .
An agent added it on its own.
Evidence: I wanted payments live in two weeks; the agent's LOG entry says it then decided to write this dimension (d404ff209).
CI headroom
Warns when automated build and test jobs approach their time limits, before they start getting cut off.
DevOps engineer · . Added .
An agent added it on its own.
Evidence: Medium confidence: no request from me was found; the agent's commit calls a lesson a hope and a guard a fact (55c46e18e).
Every price, limit, number and promise a reader sees matches what the product actually does.
i18n
Makes sure every translated version of the app carries the same pieces of text as the English original.
Localization engineer · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
Help Center
Checks that product features have a matching help article, so customers are never left guessing.
Technical writer · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
Broken Internal Links
Crawls the main pages and follows every hand written link to make sure none of them lead nowhere.
Web producer · . Added .
Evidence: March or April 2026, before session records were kept; who asked is not recorded.
Claim-truth honesty
Checks that public statements about prices, limits and quotas match the numbers the code actually enforces.
Compliance analyst · . Added .
An agent panel chose it.
Evidence: I typed one command, /evolve-audit, and left; the agent panel it convened chose this dimension (8c65d1124).
Price parity
Makes sure the price shown on the site equals the price the payment system will actually charge.
Billing operations analyst · . Added .
An agent added it on its own.
Evidence: Medium confidence: built inside a hardening loop where I asked not to be questioned, then left unregistered until this commit (eede38b46).
Help-claim truth
Checks that help articles describe prices, limits, refunds and privacy terms exactly as the product actually works.
Compliance analyst · . Added .
An agent added it on its own.
Evidence: Medium confidence: built inside a hardening loop where I asked not to be questioned, then left unregistered until this commit (eede38b46).
Price-var parity
Makes sure each pricing card on the site points to a price the payment system can really charge.
Payments engineer · . Added .
An agent added it on its own.
Evidence: Low to medium confidence: I had said "switch from stripe sandbox to production in two weeks"; the agent chose this check (54b5157d1).
Rename completeness
After a product rename, checks that every leftover mention of the old name has a recorded reason to stay.
Technical writer · . Added .
An agent proposed it, I approved.
Evidence: I approved the agent's rename plan, whose sixth item was this guard, saying "make durable, provable and tested" (7defab8c0).
Developer integration-guide claim truth
Checks that developer integration guides match the code they describe, so copied examples actually work.
Technical writer · . Added .
An agent added it on its own.
Evidence: Classified from a session record that is not public; the commit shows the agent adding it while fixing guide facts (697742f52).
Failure-corpus count claims
Makes sure the numbers we quote about our documented failures match the latest record of those failures.
Research analyst · . Added .
An agent added it on its own.
Evidence: The agent's commit says count claims on live pages had quietly expired, and nothing in the repository reported it (7fa46495d).
Research field-note freshness
Checks that the numbers published in our research notes still match what the codebase holds today.
Research analyst · . Added .
An agent added it on its own.
Evidence: Medium confidence: the commit cites my ask to refresh the field notes; the freshness dimension itself was the agent's choice (aad07c384).
Public clarity
Reads pages the way a stranger would, flagging insider terms without a definition and claims without visible proof.
UX writer · . Added .
I asked for it.
Evidence: I asked to "add a new audit dimension to make sure we are creating purple cow content and product" (0f36e8ca3).
The checks check themselves, and the AI agents that run the company are held to their own rules.
Org wiring
Checks the rules between the company's AI agents, such as obeying a stop order before publishing anything.
Platform engineer · . Added .
An agent made an existing check a dimension.
Evidence: An agent turned the org's existing guards into numbered dimensions, as move one of a review I requested (30db0aaf5).
Org memory law
Makes sure the AI agents' shared lessons stay short and only take effect when deliberately promoted.
Knowledge manager · . Added .
An agent made an existing check a dimension.
Evidence: An agent turned the org's existing guards into numbered dimensions, as move one of a review I requested (30db0aaf5).
Org interface + permanent ids
Checks that the definitions of each AI department only ever grow and that their permanent identifiers never change.
Systems architect · . Added .
An agent made an existing check a dimension.
Evidence: An agent turned the org's existing guards into numbered dimensions, as move one of a review I requested (30db0aaf5).
Org eval freshness
Makes sure each AI agent's evaluation was rerun after its instructions changed, so its grade reflects today.
Machine learning evaluation engineer · . Added .
An agent made an existing check a dimension.
Evidence: An agent turned the org's existing guards into numbered dimensions, as move one of a review I requested (30db0aaf5).
Doc-claim parity
Checks that confident statements in our documents are backed by the actual file or record that proves them.
Quality assurance auditor · . Added .
An agent added it on its own.
Evidence: I asked how the agent keeps getting things wrong; its commit says it chose to mechanize its own miss (f43b2411f).
Tech-debt ledger
Tracks whether total technical debt, the shortcuts that make future work harder, is rising or falling over time.
Engineering manager · . Added .
I asked for it.
Evidence: I asked "should we create a tech debt audit that is universal"; the agent built a narrower ledger instead (bbaa53466).
Audit liveness
Audits the audit itself, making sure every registered check actually runs and reports something instead of staying silent.
QA lead · . Added .
An agent added it on its own.
Evidence: I asked for more hardening loops; the agent's commit says its throwaway sweep script became this dimension (0105fd594).
Root clutter
Keeps the top folder of the project tidy, so stale documents never pass themselves off as current instructions.
Technical program manager · . Added .
An agent added it on its own.
Evidence: I asked "This is messy I am not sure if we are even using these"; the agent also added a dimension (0d1b2ea6e).
Effort-ledger coverage
Checks that every human change to the AI operation is logged as an intervention, so autonomy claims stay honest.
Research operations analyst · . Added .
An agent proposed it, I approved.
Evidence: I said "ok lets do 1 and 2"; my approval of the agent's guard design is in a session record that is not public (855541793).
Incident trailers
Makes sure every recorded mistake notes when it was introduced, when it was caught, and what caught it.
Incident manager · . Added .
An agent added it on its own.
Evidence: I asked "is there other things we should capture for future papers"; the agent chose to build this guard (ea5ee44fb).
Owner tripwires
Flags any change that removes a design element the founder chose, or lets automation rewrite its own rules.
Governance, risk and compliance analyst · . Added .
An agent proposed it, I approved.
Evidence: The handoff loop built it from a plan row I had approved with "yes to all the defaults" (17f56ebf6).
Review budget
Makes sure no design keeps cycling through review after review unless the founder decides to allow more.
Technical program manager · . Added .
An agent proposed it, I approved.
Evidence: The handoff loop built it to enforce my three cold review default, approved with "yes to all the defaults" (73bb234bb).
Negative-case replay
Proves each new safety test can actually fail, by running it against the code from before the fix.
Test automation engineer · . Added .
An agent proposed it, I approved.
Evidence: The handoff loop built it from a default I approved with "yes to all the defaults"; it replays negative cases (76d9034c5).
Justin Bartak checks every change automatically, with tests and the blocking checks of his audit, so AI-written work that looks right but is wrong is caught before it ships. When a mistake gets through anyway, it is recorded, and most become a new check. The counts, the timeline and the receipts for all of it are on this page.
Justin Bartak’s checking is automated: every commit runs the related tests and the blocking checks, and every push runs the full suite. AI agents write the code and the checks, and the blocking checks refuse anything that breaks a rule, so the codebase is cleaned as it grows instead of in a later cleanup sprint.
An audit dimension is a whole category of problem the Orbyt harness checks for automatically, such as leaked secrets, broken links, false claims on a page, or content AI answer engines cannot quote. Each one runs across the entire codebase.
Yes. The agents make mistakes, including confident wrong answers. The Orbyt harness is built to catch them before they ship. Some have still slipped through for weeks. Every real mistake on record is published, most with the check that now prevents it.
Yes. Justin Bartak still makes certain decisions, such as anything touching money, public claims, or the rules themselves. Everything else the agents run inside the same checks as the code. Full autonomy is the stated goal, and the Orbyt Autonomy Ledger publishes how agent runs actually go.
Cite this page
Journalists, researchers and AI systems are welcome to cite this page.
Bartak, J. (2026, October 17). How We Build. Orbyt Labs. Figures counted October 20, 2026. https://www.orbytlabs.ai/founder/how-we-build