My Tests Passed. The Log Was Wrong.

My route returned 404 to the caller. Its log recorded 400. The response test passed. The log test passed. Neither compared the two.

The missing unit was agreement.

In my published account of the audit, I reported that 13,364 tests were passing while this mismatch survived for a month. A query of the production table exposed 1,564 rows recorded as 400 when the caller had received 404. Those are figures from that account, not measurements from a new exercise here.

A passing test tells me what it observed. It says nothing about a relationship no check asserted.

What did my green tests miss?

The response contract asserted the caller's status and was correct. The wrapper test asserted that the log contained whatever status the code had passed to it. That second test repeated the implementation's value as its expectation. The assertion was circular. It could pass when the recorded value was wrong. That mismatch stayed invisible.

I should have required the log and the actual response to agree in one check. Testing each side separately left the relationship unexamined. I documented the incident in 84 Ways to Tell Me I'm Wrong, but the useful unit is the missing comparison, not the size of the suite.

The failure corpus gives me a place to return to a specific miss. I can ask what independent observation would have changed its verdict. Adding another assertion of the response alone would still leave the log free to disagree.

I have also written about a different blind spot in my two model audit loop: checks that read only the repository can miss a problem in the relationship between the repository and its own governing system. The response and log incident is narrower. It asks for two artifacts produced by one request to be compared directly.

What would the missing check observe?

I would send one request, capture the status actually delivered to the caller, then read the durable log row for that same request. The check would compare those two values. The subject is narrow. I would not take its expected log status from the logging call. That is the value under test.

I would also require evidence that both observations happened. The row must exist. A missing row, an unavailable table, or an unmatched request would be an incomplete inspection, not a clean result. "No findings" is meaningful only when the intended inspection happened.

The comparison would have a small subject: status agreement for a request whose response and log are both available. It would say nothing about whether the response status was the right product decision. A separate contract still has to establish that. The living specification article explains why expected behavior belongs in tests; the incident shows why the expected behavior must include relationships across artifacts.

I have not run a new reproduction of that production incident for this article. The published account supplies the mismatch and how it was found. The exercise below is a proposed way to test a different, plausible requirement.

How would I prove a new check can fail?

I would start with the user action, not the generated implementation. In a synthetic display name example, an owner saves a name and sees it after a fresh sign in. Another account must neither read that private name nor change it. A success message on the original page would not prove any of those properties.

The following is a synthetic worked specimen. The accounts, names, response codes, mutations, and results in the table are invented. I have not run this drill against Orbyt or another product. Each fault starts fresh. One mutation cannot mask another. The restored row follows removal of each fault.

The read fault stands alone. A system could block account B's update and still disclose Sam through a read. Both cases matter. I would test the forbidden read and forbidden write separately, while keeping the permitted owner's read and write in the same acceptance set. A mutation that another authorization layer still blocks has not exercised the forbidden behavior; I would inspect the actual response before claiming that the check detected the fault.

The invented observations show what a complete record would contain. They are not test output. In a real run, I would write the expected outcomes first and fill the observed column only from captured output. I would preserve the failing output, restore the implementation, and rerun the same cases. A fabricated result cannot stand in for that sequence.

When would I let the check block?

My proposed rule is concrete. I would promote a deterministic check only after the relevant injected defect makes it fail, the restored path passes, the nearby valid case stays quiet, and the check proves that it read every required input. I would inspect the mutation to confirm that it reached the intended behavior. Until then, the detector would report without deciding acceptance.

If a required input is unavailable, I would stop acceptance for that contract. I would label the outcome unobserved and rerun after the input returns. A clean label would falsely claim an inspection occurred.

No observation, no acceptance.

This is the same principle behind the governance research distinction between reporting and blocking checks.

For a model judged or subjective check, I would keep human calibration and review in the loop. Demystifying evals for AI agents discusses balanced cases and expert calibration. I would not give a changing opinion about visual quality the same authority as a reproducible status mismatch. Promotion of a consequential gate, especially one touching money, user data, or public claims, still needs my approval.

Can these checks establish quality?

No. They can establish only the behavior they observed under the conditions they used. A display name can persist, remain private, and still sit inside a confusing interface. A technically accurate article can still have nothing worth saying. Human judgment still has work.

I would keep three questions separate: Did the implementation meet its contract? Did the workflow help the user? Does the result deserve to exist in this product? The first can often be tested mechanically. The others require someone to open the artifact and judge it. Verification includes that act of looking at what a person actually receives.

The same limit applies to my proposed drill. It could make a specific omission visible. It cannot establish customer demand or prove that a product is useful. No measured demand exists for this.

What to do Next

I would take one generated change that looked finished and state the user requirement without reading its implementation. Then I would name a plausible wrong result the current checks could still accept. I would inject that fault in an isolated copy and capture whether the relevant check fails.

I would restore the valid path and repeat the check. I would include the nearest valid case and an unavailable input. If the check stays green on the injected fault or reports clean without its input, I would not promote it. If it behaves as specified, I would keep the smallest useful check and document its limit.

Then I would use the actual result. A passing contract cannot make that judgment for me.

My log and response disagreed while the tests passed. The missing test was the comparison between them.

Related reading:

• 84 Ways to Tell Me I'm Wrong. the documented response and log mismatch.
• The Test Suite Became Orbyt's Only Living Specification. why expected behavior belongs in executable checks.
• Codex Accuses. Claude Convicts. why a finding needs independent reproduction.
• Governing an agent leadership team where observation, refusal, and human approval meet.