An agent that reports success has made a claim. Observability is the practice of checking that claim against a record the agent did not write. On the pulse of , all nine of its public axes were observed, and three were red. Its command axis carried nine records whose status was not observed, said as such rather than counted as clean.
The daily pulse does that checking, and the pulse is a script with no model in it. 74 of the rows in the AI Oversight Ledger come from its findings, which makes it the monitor behind most of the 75 catches monitors found. The founder's own catches are NOT OBSERVED in that ledger, so this is not a share of all catches.
What is agent observability at Orbyt Labs?
Checking what an agent says it did against a record it did not write. The instrument is the daily pulse: a script, scheduled every day at 19:00 UTC, that reads GitHub's run records, the live cron table, the committed artifacts and the command records, and reports what is open and what is red.
No model writes any part of it. The reason is in the script's own header: a pulse that cannot call a model cannot invent a figure.
Most of what the AI Oversight Ledger holds was caught this way. It files every finding the pulse raises under a monitor, and monitors account for 75 of the 97 catches it holds through .
What can this page not see?
The limits come first.
- One organization, one repository, one founder. There is no control group.
- The pulse sees what leaves a record: GitHub runs, committed files, the cron table, command records. An action that leaves no record is invisible to it.
- The public pulse file carries the axes that are safe to publish. The private report carries more, and their verdicts are not on this page.
- The founder's own catches are NOT OBSERVED in the ledger, so a monitor's count is not a share of all catches.
- Many monitor catches are a workflow that failed, which a monitor sees by design: 39 of the 97 rows are workflow failures, and 27 were seeded, opened by a pulse that raised five or more findings at once.
- Who cancelled a run is NOT OBSERVABLE here: GitHub's audit log, which would say, covers organizations only, and this repository is personal.
- No latency, step timing or token trace is measured on this page.
What does an instrument say when it cannot see?
NOT OBSERVED. Every public axis ends in one of six verdicts, and one of them, NOT OBSERVED, means the pulse could not read it. A crashed checker is not a clean checker, and a query that failed is not a query that found nothing.
The rule exists because each of those confusions shipped once:
- A run-truth checker that crashed rendered as clean. A missing permission, its error thrown away, read as a tool being unavailable. A page of the newest sixty runs reported one red run of a true eleven.
- The pulse printed a monitor as undated in every pulse from , while the workflow feeding it succeeded three times a week. A cron check read the wrong field of a JSON column and matched 0 of 120 rows for 28 days.
- A monitoring query took 200 rows of a 24 hour window holding 9,072, with no filter, found none of the 18 real failures, and printed OK.
- The pulse itself failed in its commit step twice and went 51 hours stale, and nothing outside the pulse job watched its files.
Every public axis of the pulse, day by day
Three of the pulse's nine axes are red on : Did the org produce anything, Red runs in the last day and Crons running and failing.
Clean
Open founder directives
Red
Did the org produce anything
Clean
Monitors nobody fed
Red
Red runs in the last day
Red
Crons running and failing
Warning
API command attribution
Clean
Halts nobody commanded
Clean
Does the record still reproduce
Clean
Open pull requests sitting red
Did the org produce anything: red on the newest pulse, and red on 27 of the last 28 committed pulses.
RedWarningCleanNo reading: no committed pulse
Point at a tile, or focus the board and use the arrow keys.
Tap a tile to read its red count above.
- Clean means, in the pulse's own words: The observed portion of the axis had no findings. Partial reads and stated limits still apply.
- A day with no committed pulse is hatched and counted as no reading. It says that no revision of public/pulse.json carries that day, not that the pulse did not run.
Not shown. Each axis's items and counts, which the daily pulse page prints for the newest pulse, and the axes of the private report.
The newest day: public/pulse.json. The days before it: pulse-history.generated.json, the newest revision of public/pulse.json for each day in its git history, exported through by scripts/research/method-figures/agent-observability.ts.
Pulse of
The numbers as a table
| Axis | Newest pulse | Red days | Warning days | Clean days | No reading days |
|---|---|---|---|---|---|
| Open founder directives | Clean | 0 | 0 | 28 | 2 |
| Did the org produce anything | Red | 27 | 0 | 1 | 2 |
| Monitors nobody fed | Clean | 6 | 0 | 22 | 2 |
| Red runs in the last day | Red | 28 | 0 | 0 | 2 |
| Crons running and failing | Red | 15 | 0 | 13 | 2 |
| API command attribution | Warning | 0 | 28 | 0 | 2 |
| Halts nobody commanded | Clean | 0 | 0 | 28 | 2 |
| Does the record still reproduce | Clean | 2 | 0 | 26 | 2 |
| Open pull requests sitting red | Clean | 4 | 0 | 24 | 2 |
Download the data: agent-observability-pulse-board.csv, agent-observability-pulse-board.json
A blind instrument that reads as healthy is worse than no instrument. The count it prints is the first thing to distrust when it equals a page size.
How is one run traced to its outcome?
By six links, each read from a source other than the agent's report: the stats row, what invoked the run, the transcript's own result event, the configuration hash, the commit that landed the row, and GitHub's own conclusion. Each link is proven, contradicted, not observed or not applicable, and the run's verdict is the worst of them.
The pulse reports two of the six each day it runs: the configuration and the landing. On , 166 committed run rows carried a configuration hash, 166 of them reproduced from git at their own revision, and the landing commit was found for 165, with one not applicable.
The sweep reads committed run rows. A run that never wrote a row is not in it, and the pulse's run axes have to catch it instead.
Who watches the watcher?
A second check outside the pulse job. Since lesson 93 a freshness watchdog runs in the ladder workflow and alarms when the pulse's committed files pass a 30 hour ceiling. A read that fails is NOT OBSERVED with its reason, never fresh.
It is not fast. Lesson 93 records that it alarms 36 to 40 hours after the last good commit.
What happens when the pulse finds something red?
It becomes a repair trigger. A pulse run that failed, was cancelled, timed out or never started is itself a trigger, and the run read is the newest completed one whose pulse job actually ran, because a second nightly schedule in the same workflow skips the pulse job and would otherwise hide a failure.
The repair path, pulse-fix, is designed to run on the founder's Mac every 20 minutes through the afternoon, in a clone whose push URL is disabled, inside a fence, with a turn cap. It lands only a verified commit, by fast-forward, through a gate it runs itself. When a scan finds a secret, a planted instruction or a change to a file that judges the work, or cannot read what it scans, nothing is published and the commits are kept in a local quarantine. A commit refused for any other reason goes to a pull request instead.
Its record is thin. As of , two commits on main carry the claim trailer: one pulse-fix landed during a test walk, which says it has nothing a later pulse could check, and a negative control an agent wrote in the founder's session.
Whether pulse-fix is scheduled on the founder's Mac today is NOT OBSERVED from the repository. A claimed repair is judged by a later pulse from the workflow's own runs, never by the fixer, so green means verified and never that an agent said so.
How is this different from an LLM tracing tool?
A tracing tool records the steps inside a run: the calls, the tool uses, the timings. This method asks a different question at the end of the run: did the outcome the agent reported happen, according to a record the agent did not write?
The two answer different questions, and one does not replace the other. This page makes no claim about any tracing product's features.
Agent observability questions? Answers.
Related reading
- AI oversight ledger
The standing record of who caught each issue and who fixed it.
- Autonomous operation
The dated record of the Machine running without a person in the loop.
- The kill switch
How the Machine is stopped, with the dated record of its halts.
- Harness engineering
How agent mistakes become standing instructions, tests, checks and gates.
- The daily pulse
Today's pulse, published as it ran, with every axis and its verdict.
- All Orbyt Labs methods
Every method page, each with its dated public record.
Cite this page
Cite this page for the practice and follow its sources for the underlying records.
Bartak, J. (2026). Agent Observability. Orbyt Labs Research. https://