--- title: "I Built an AI Visibility Probe. It Said Zero." description: "This is not a study. There is no holdout cohort, the sample is one company, and in 40 runs that returned answers, through Nov 9, 2026, my probe of one model recorded no answer naming Orbyt and, in the two runs that counted them, no citations of Orbyt pages. The useful part is not the number. It is that my instrument lied to me three times, and every lie took the shape of a confident report." author: "Justin Bartak" published: 2026-10-15 modified: 2026-11-09 canonical: https://www.orbytlabs.ai/blog/measuring-ai-citations --- # I Built an AI Visibility Probe. It Said Zero. I built an instrument to see whether one AI model surfaces Orbyt when someone asks a question in its space. Here is the complete result. **Zero. No answer named Orbyt in 40 answered runs, and the two runs that counted source citations found none.** Read that line narrowly. Every other run counts only whether an answer names Orbyt. They are not citation counts, although this post used to call them that. Only the Sep 17 and Sep 18 runs count source citations, each with just 3 answers that returned a web source. None of it is a count of how often AI engines cite me. My Bing Webmaster Tools AI Performance report for orbytlabs.ai, which lists its citation sources as Microsoft Copilots and Partners, showed 35.4K citations from Aug 25 to Nov 6, 2026. The two instruments measure different scopes, and this post is about the first. Before you decide whether that is interesting, three disclosures, because this post is worth nothing without them. This is not a study. There is no holdout cohort, because I optimized every page on the site, which means I have no control group and never will unless I deliberately stop optimizing something. The sample is one company, one vertical, one author. And the instrument measures one model's behavior on my queries, which is a much smaller claim than "what gets cited." **It is an observation. I am titling and framing it as one, because calling it a study would be the exact dishonesty the post is about.** ## The hypothesis, stated before the data I want this on the record in the order it happened. My bet was that a site with genuinely first-party data, answer-shaped paragraphs, and a clean entity graph would get cited in AI answers, and that I would be able to watch it happen. I put real work behind that bet: a statistics page rendering from a single source of truth, a deposited dataset with a DOI, structured author identity, markdown twins of every article. Then I built the probe so the bet could be wrong in public. ## What is the probe, precisely? A fixed set of questions in Orbyt's space, asked of one model tier, such as "What's the best app to track my job applications?" A trained run attaches no tools, so it reads what the model already knows. It has no sources to count. A retrieval run lets the model search the web. Each run records whether the answer names Orbyt or one of its domains. Since Sep 17, 2026 a retrieval run also records whether a returned source URL sits on an Orbyt domain, which is the only thing here that counts as a citation. The result is appended to a history file so a trend can exist. Here is the entire history. | Run | Mode | Answers naming Orbyt | Answers with a web source | Orbyt sources cited | | --- | --- | --- | --- | --- | | 22 runs, 2026-07-12 to 2026-09-16 | Trained | 0 of 25 to 56 | none possible | not measured | | 2026-08-06 | Retrieval | 0 of 8 | not recorded | not measured | | 2026-09-17 | Retrieval | 0 of 56 | 3 of 56 | 0 | | 2026-09-18 | Retrieval | 0 of 65 | 3 of 65 | 0 | | 15 runs, 2026-10-05 to 2026-11-09 | Trained | 0 of 66 to 81 | none possible | not measured | | 2026-11-04 | Trained | no answers returned | none possible | not measured | On Nov 4 every query hit the model's usage limit, and the run still finished green. ## The instrument lied to me three times This is the part I would keep if you only read one section, because it generalizes far past search. **The probe ran green for weeks while producing nothing.** The workflow commits nothing to the repository on purpose, because a bot push three times a week burns three production builds a week against a cost rule I care about. So it uploaded its result to a run artifact and walked away. The consequence nobody had traced: the history file, whose entire purpose is to accumulate a trend line, **could never hold more than two entries.** Every run checked out the repository copy frozen at its original baseline of one entry, appended today, uploaded two, and discarded them. The next run started from the same frozen one. Roughly eight runs measured the metric I care most about and threw the reading away. **A metric that is measured and then discarded is indistinguishable from a metric that was never measured.** Then the second lie. I fixed that by having each run inherit the previous run's artifact. The fix **had never once worked.** The restore step called the GitHub CLI in a workflow with no explicit permissions block. The default token is read-only, so the query returned 403. The step discarded stderr. The fallback converted the failure into an empty result, which is exactly what a legitimate first run produces. So the workflow concluded "no previous run" every single time. It reported success. Two separate failures. Both silent. Both shaped like a green check mark. I had to fix the instrument twice. Then the third, and I repeated this one in public. The checker's first version filed every answer that named Orbyt under a field called cited and printed CITED beside it. A name in an answer is a mention. The code split the two on Sep 17, 2026. This post kept calling them citations until Nov 5, 2026. I was wrong for seven weeks after the code knew better. Another AI read it and repeated it. **Before you trust any number about your AI visibility, prove your instrument can fail.** This is the same wall I hit trying to make [a self-improving system grade itself](https://www.orbytlabs.ai/blog/recursive-self-improvement-ceiling). A loop that generates its own evidence will hand you a clean report forever. The only fix is an oracle standing outside it. ## The one place external measurement did work I want to be fair to instrumentation, because the story so far makes it sound futile. It is not. It is just that the useful signal came from somewhere I was not looking. Search console query data, from a completely different engine, told me something no amount of internal checking could. It showed that of 61 job titles people actually searched for, 50 had no page on the site at all, while thousands of pages existed for titles nobody searches. Every internal guard reported the system healthy, because the page generator and the sitemap agreed with each other perfectly. **Internal consistency is not evidence. Two of your own systems agreeing proves only that they share an assumption.** But that data carries a bias. **A page that returns 404 cannot earn an impression.** So demand data gathered through a gap is censored: it can only ever tell you about queries that partially worked. It reads as a floor on what you lost, never an estimate of it. **When your instrument can only see through the hole, do not use its output to decide the shape of the hole.** The same trap sits inside my probe, and I have not solved it. Questions I chose, about topics I decided Orbyt should surface for. If the real answer is that nobody asks questions in that shape, my probe cannot tell me. It can only report that one model did not surface Orbyt for the questions I already believed mattered. ## What this cannot show Stated plainly, because the value of the post is in this section. **No control group.** Every page was optimized. Without a holdout cohort I cannot separate the effect of the work from the effect of time, of the domain aging, or of the engines changing. If the number moves next quarter, I will not know why. That is the cost. **No causal claim, ever, in this design.** Even a rising citation count would be correlational at best. I would be one site watching one number. **Single everything.** One vertical, one company, one author, one model tier sampled on 40 dates. Answer engines differ enormously in how they retrieve, and one model's behavior generalizes to the others roughly as well as one browser's rendering generalizes to all browsers. **Small denominators.** 25 questions, then 8, 56 and 65, and only 3 answers on each of the last two dates returned a source. On 25, a single well-placed page could move the rate by four points and mean nothing. On 3, a citation rate is barely a rate. ## What do I actually believe, and on whose evidence? I trust other people's numbers here more than my own, because theirs rest on samples mine cannot approach. Off-site brand mentions correlate with AI visibility far more strongly than backlinks do, by roughly three to one in an [Ahrefs](https://ahrefs.com) study across tens of thousands of brands. That finding changed what I do. The lever is outside your domain. Self-contained answer capsules under question headings appear in the large majority of cited pages in published audits, and capsules without links inside them do better. Original statistics lift generative-engine visibility measurably in the academic work. AI-cited URLs skew fresher than the organic top ten. And the finding that cost me the most to accept: **llms.txt appears to be essentially unread.** A twelve-week server-log study found single-digit fetches from major AI crawlers against thousands of robots.txt requests. I keep mine as cheap insurance and I no longer count it as work. Nothing lives there that is not also on an indexed page. I have written the practical version of all of this in [the GEO and AEO playbook](https://www.orbytlabs.ai/blog/aeo-geo-llms-txt-ai-traffic). What is new here is that my own probe does not yet confirm any of it. ## The practical position Instrument first. Then invest only in what survives a ranking change. That second clause is the whole strategy. So the test I apply before doing any of this work is simple: **if the retrieval landscape reshuffles completely next quarter, is this still worth having done?** First-party data you own passes. A dataset with a DOI passes. Clear answers to real questions pass. Honest freshness passes. Being mentioned by other people passes. Schema micro-optimization does not pass. Chasing a specific crawler's token does not pass. Stuffing a file nobody fetches does not pass. ## What to do Next Build the probe before you build the strategy. A fixed query set, a schedule, a recorded result. It will take an afternoon and it will be the only thing standing between you and a year of confident, unfalsifiable work. Then try to make it report a failure. Break it on purpose and confirm it says so. If it cannot tell you the difference between "zero citations" and "the check did not run," you have a decoration. Then pick a baseline date and write down your hypothesis before you look. In public if you can stand it. I would rather publish a zero with a documented method than a rising number with no baseline. One of those is evidence and the other is a story, and the difference only shows up when someone asks you how you know. **The probe found no answer that named Orbyt. That is a worse number and a better position than not having a probe.** Related reading: - [AEO, GEO, and llms.txt: The New SEO](https://www.orbytlabs.ai/blog/aeo-geo-llms-txt-ai-traffic) the practical playbook this post is the honest measurement of - [AEO: No One Has the Answer Key](https://www.orbytlabs.ai/blog/aeo-no-one-has-the-answer-key) why the confident advice in this space outruns the evidence - [84 Ways to Tell Me I'm Wrong.](https://www.orbytlabs.ai/blog/ai-code-audit-harness) the same instrument-honesty rule applied to code - [Verification Is the New Literacy](https://www.orbytlabs.ai/blog/verification-is-the-new-literacy) checking the instrument is the skill, not reading the output - [Built to Be Quoted.](https://www.orbytlabs.ai/blog/built-to-be-quoted) the passage engineering behind the bet - [Citations. Citations. Citations.](https://www.orbytlabs.ai/blog/citations-citations-citations) share of answer, a wider measure than this probe This is the markdown version of https://www.orbytlabs.ai/blog/measuring-ai-citations. The page there is the canonical copy.