Agent Observability

What an agent claims, checked against records it did not write.

The AI Oversight Ledger holds 97 catches, and 75 were found by a monitor. How the pulse watches, blind spots included.

By Justin Bartak. Updated .

An agent that reports success has made a claim. Observability is the practice of checking that claim against a record the agent did not write. On the pulse of , all nine of its public axes were observed, and three were red. Its command axis carried nine records whose status was not observed, said as such rather than counted as clean.

The daily pulse does that checking, and the pulse is a script with no model in it. 74 of the rows in the AI Oversight Ledger come from its findings, which makes it the monitor behind most of the 75 catches monitors found. The founder's own catches are NOT OBSERVED in that ledger, so this is not a share of all catches.

What is agent observability at Orbyt Labs?

Checking what an agent says it did against a record it did not write. The instrument is the daily pulse: a script, scheduled every day at 19:00 UTC, that reads GitHub's run records, the live cron table, the committed artifacts and the command records, and reports what is open and what is red.

No model writes any part of it. The reason is in the script's own header: a pulse that cannot call a model cannot invent a figure.

Most of what the AI Oversight Ledger holds was caught this way. It files every finding the pulse raises under a monitor, and monitors account for 75 of the 97 catches it holds through .

What can this page not see?

The limits come first.

  • One organization, one repository, one founder. There is no control group.
  • The pulse sees what leaves a record: GitHub runs, committed files, the cron table, command records. An action that leaves no record is invisible to it.
  • The public pulse file carries the axes that are safe to publish. The private report carries more, and their verdicts are not on this page.
  • The founder's own catches are NOT OBSERVED in the ledger, so a monitor's count is not a share of all catches.
  • Many monitor catches are a workflow that failed, which a monitor sees by design: 39 of the 97 rows are workflow failures, and 27 were seeded, opened by a pulse that raised five or more findings at once.
  • Who cancelled a run is NOT OBSERVABLE here: GitHub's audit log, which would say, covers organizations only, and this repository is personal.
  • No latency, step timing or token trace is measured on this page.

What does an instrument say when it cannot see?

NOT OBSERVED. Every public axis ends in one of six verdicts, and one of them, NOT OBSERVED, means the pulse could not read it. A crashed checker is not a clean checker, and a query that failed is not a query that found nothing.

The rule exists because each of those confusions shipped once:

  • A run-truth checker that crashed rendered as clean. A missing permission, its error thrown away, read as a tool being unavailable. A page of the newest sixty runs reported one red run of a true eleven.
  • The pulse printed a monitor as undated in every pulse from , while the workflow feeding it succeeded three times a week. A cron check read the wrong field of a JSON column and matched 0 of 120 rows for 28 days.
  • A monitoring query took 200 rows of a 24 hour window holding 9,072, with no filter, found none of the 18 real failures, and printed OK.
  • The pulse itself failed in its commit step twice and went 51 hours stale, and nothing outside the pulse job watched its files.

Every public axis of the pulse, day by day

Three of the pulse's nine axes are red on : Did the org produce anything, Red runs in the last day and Crons running and failing.

Clean

Open founder directives

Red

Did the org produce anything

Clean

Monitors nobody fed

Red

Red runs in the last day

Red

Crons running and failing

Warning

API command attribution

Clean

Halts nobody commanded

Clean

Does the record still reproduce

Clean

Open pull requests sitting red

Did the org produce anything: red on the newest pulse, and red on 27 of the last 28 committed pulses.

RedWarningCleanNo reading: no committed pulse

Point at a tile, or focus the board and use the arrow keys.

Tap a tile to read its red count above.

One tile per public axis of the Orbyt Daily Pulse, in the pulse's own order. The shape and the word on a tile are the axis's verdict on the newest pulse; the strip under its name is the last 30 calendar days, one tick a day, the newest on the right.
  • Clean means, in the pulse's own words: The observed portion of the axis had no findings. Partial reads and stated limits still apply.
  • A day with no committed pulse is hatched and counted as no reading. It says that no revision of public/pulse.json carries that day, not that the pulse did not run.

Not shown. Each axis's items and counts, which the daily pulse page prints for the newest pulse, and the axes of the private report.

The newest day: public/pulse.json. The days before it: pulse-history.generated.json, the newest revision of public/pulse.json for each day in its git history, exported through by scripts/research/method-figures/agent-observability.ts.

Pulse of

The numbers as a table
Every public axis of the pulse, day by day
AxisNewest pulseRed daysWarning daysClean daysNo reading days
Open founder directivesClean00282
Did the org produce anythingRed27012
Monitors nobody fedClean60222
Red runs in the last dayRed28002
Crons running and failingRed150132
API command attributionWarning02802
Halts nobody commandedClean00282
Does the record still reproduceClean20262
Open pull requests sitting redClean40242

Download the data: agent-observability-pulse-board.csv, agent-observability-pulse-board.json

A blind instrument that reads as healthy is worse than no instrument. The count it prints is the first thing to distrust when it equals a page size.

How is one run traced to its outcome?

By six links, each read from a source other than the agent's report: the stats row, what invoked the run, the transcript's own result event, the configuration hash, the commit that landed the row, and GitHub's own conclusion. Each link is proven, contradicted, not observed or not applicable, and the run's verdict is the worst of them.

The pulse reports two of the six each day it runs: the configuration and the landing. On , 166 committed run rows carried a configuration hash, 166 of them reproduced from git at their own revision, and the landing commit was found for 165, with one not applicable.

The sweep reads committed run rows. A run that never wrote a row is not in it, and the pulse's run axes have to catch it instead.

Who watches the watcher?

A second check outside the pulse job. Since lesson 93 a freshness watchdog runs in the ladder workflow and alarms when the pulse's committed files pass a 30 hour ceiling. A read that fails is NOT OBSERVED with its reason, never fresh.

It is not fast. Lesson 93 records that it alarms 36 to 40 hours after the last good commit.

What happens when the pulse finds something red?

It becomes a repair trigger. A pulse run that failed, was cancelled, timed out or never started is itself a trigger, and the run read is the newest completed one whose pulse job actually ran, because a second nightly schedule in the same workflow skips the pulse job and would otherwise hide a failure.

The repair path, pulse-fix, is designed to run on the founder's Mac every 20 minutes through the afternoon, in a clone whose push URL is disabled, inside a fence, with a turn cap. It lands only a verified commit, by fast-forward, through a gate it runs itself. When a scan finds a secret, a planted instruction or a change to a file that judges the work, or cannot read what it scans, nothing is published and the commits are kept in a local quarantine. A commit refused for any other reason goes to a pull request instead.

Its record is thin. As of , two commits on main carry the claim trailer: one pulse-fix landed during a test walk, which says it has nothing a later pulse could check, and a negative control an agent wrote in the founder's session.

Whether pulse-fix is scheduled on the founder's Mac today is NOT OBSERVED from the repository. A claimed repair is judged by a later pulse from the workflow's own runs, never by the fixer, so green means verified and never that an agent said so.

How is this different from an LLM tracing tool?

A tracing tool records the steps inside a run: the calls, the tool uses, the timings. This method asks a different question at the end of the run: did the outcome the agent reported happen, according to a record the agent did not write?

The two answer different questions, and one does not replace the other. This page makes no claim about any tracing product's features.

Agent observability questions? Answers.

Checking what an agent reports against a record it did not write, and saying NOT OBSERVED wherever the check cannot see.

So it cannot invent a figure. Every line it prints is derived mechanically from records.

On the AI Oversight Ledger, mostly monitors: 75 of 97 catches through . The founder's own catches are not observed there.

It starts a repair trigger. The repair path is designed to land only verified commits; whether it is scheduled today is not observed.

No. Tracing records the steps inside a run. This checks the outcome at the end of it, against records the agent did not write.

Cite this page

Cite this page for the practice and follow its sources for the underlying records.

Bartak, J. (2026). Agent Observability. Orbyt Labs Research. https://www.orbytlabs.ai/research/agent-observability