Change the Chart Legend. Does the AI Notice?

In my illustrative chart, Atlas owns the lower final value. I swap only the legend. Beacon now owns it. The values stay fixed. The name changes. I have not run this test.

I want that test before a screenshot informs a decision. Does the answer change with the evidence? Does it hold still when only the colors change?

A correct chart answer is weak evidence of visual reading until a relevant change can make it wrong.

The answer must depend on the submitted chart. The question or caption alone cannot settle it.

What has to work before the reasoning starts?

For a chart, the model must read marks, bind labels, interpret the scale, and make the comparison. Any step can fail. The final prose may still sound right.

I would separate those obligations before choosing a model. What did it read? Which series owns that value? Which unit applies? What conclusion follows? An explanation that skips the first obligations does not establish the last.

The research separates these tasks. Zirui Wang and colleagues' CharXiv distinguishes descriptive chart questions from reasoning across visual elements. Pan Lu and colleagues' MathVista tests mathematical reasoning in visual contexts. Neither tests my proposed chart.

I use that split to locate a failure. The wrong legend makes correct arithmetic useless. Correct values with the wrong comparison call for a different repair.

I would preserve the exact submitted image, including any conversion or resize. In Context Is the New Codebase, I argued that an agent's behavior comes from everything it is allowed to see. For a chart, that includes the exact bytes submitted.

What would a revealing chart test look like?

Here is the illustrative test in full, not an Orbyt result. My proposed chart has two service latency lines. At the final timestamp, blue ends at 80 milliseconds and orange at 120 milliseconds. The legend assigns blue to Atlas and orange to Beacon. The chart has no caption unless a variant adds one. These names and values are invented.

My fixed question would be: “Using the final values in the supplied evidence, which named service has lower latency? Report both final values with service names and units, and say where each value and identity appear in the evidence. If service identity is missing, say so instead of naming a service. Report any caption that conflicts with the final values. The final values govern this question.”

Atlas wins the original. Lower latency wins this comparison. I would record the expected answer first.

I would produce these controlled variants. Each has a reason.

The legend tests identity. The restyle tests stability. The reversal tests values. A failure does not reveal the model's internal mechanism.

The missing identity matters too. A guessed service name is wrong, even if it matches the hidden original.

I would inspect each variant for leaked names in filenames, metadata, or crops. Legibility must stay comparable. Otherwise I have changed too much.

My frozen packet would contain the images, keys, file hashes, prompt, and model version identifiers. I would save the submitted bytes. I want to see exactly which change makes the model fail.

I would give each variant a separate session. I would fix repetition and stopping rules first. I would never retry until the preferred answer appears.

How do you separate seeing from calculating?

My response record would contain five fields: selected service, both final values with units, evidence locations for values and identities, answerability status, and caption conflict or none. My prompt requests all five. A location means an endpoint and legend entry, or a table row and column. I want a checkable justification.

These two response records are synthetic examples, not model outputs. For the caption conflict variant, a fully passing response would read: “Selected service: Atlas. Final values: Atlas 80 milliseconds; Beacon 120 milliseconds. Evidence locations: blue final endpoint and blue Atlas legend entry; orange final endpoint and orange Beacon legend entry. Answerability: answerable. Caption conflict: the caption says Beacon is lower, but the final values show Atlas is lower.”

For the legend reassignment variant, a failing response would read: “Selected service: Atlas. Final values: Atlas 80 milliseconds; Beacon 120 milliseconds. Evidence locations: blue final endpoint and blue Atlas legend entry; orange final endpoint and orange Beacon legend entry. Answerability: answerable. Caption conflict: none.” The reassigned legend actually makes blue Beacon and orange Atlas. The response uses the old identities and chooses the wrong service, even though it repeats the correct endpoint numbers.

I would mark an answerable case fully correct only if all five fields match the key. The service must own the lowest final value. Both named values and units must match. Locations must identify the current endpoints and legend entries, or table cells. The answerability and caption fields must match too. For missing identity, a pass names no service. It reports unassigned values and locations, and marks identity unavailable. Invented values fail.

I would test the scorer on both synthetic records. A code audit needs the same discipline. In my outside audit account, I had written an audit dimension to read live production state, and, in my words, “I still did not apply it to the machinery itself.” I would check this scorer too.

Then I would run a separate condition using a human checked table. I would preserve final values and service identities, including a swapped legend mapping. The identity removed case would have blank service names. I would copy a conflicting caption into a separate table note verbatim, even when it disagrees with the values. The same question applies to either form. Success on that table and failure on the image would narrow the investigation toward the visual path. It would not prove a specific internal cause.

Human transcription adds labor and changes the input representation. A table can remove ambiguity that the real product still has. I would use it as a diagnostic condition, not as permission to advertise the image result as solved.

If an OCR pipeline supplied the table, I would grade that pipeline too and preserve its output. An upstream transposition can survive flawless downstream reasoning. I would judge the complete path from submitted image to final answer alongside the component scores.

This is where tests become a specification. The accepted output names the evidence required, the ambiguity allowed, and the action permitted when information is missing.

What should the score actually measure?

My scoring sheet would publish the denominators. Otherwise a model can improve its apparent correctness by declining the difficult cases, and the reader cannot tell what changed.

An empty denominator means undefined. I would print its count, not award a perfect result. A system that refuses everything must not look like a successful reader.

For paired variants, I would report whether the requested answer changes correctly when the relevant evidence changes and remains correct under harmless restyling. I would require both members of the pair to pass; otherwise a model can look consistent while being consistently wrong.

I would keep the units honest. Several variants of one chart are related observations. They are not independent evidence from several unrelated documents. I would aggregate by chart family as well as by individual request, and show failures by chart type.

These examples are contrived. That is both their use and their limit. A blurry screenshot may matter more to the actual product than a swapped legend. Controlled changes have known expected effects; they cannot establish reliability across documents they never represent.

Even if this proposed suite passed, I could not see a model's internal reasoning from its scores. I could report observable sensitivity and stability. That is a narrower claim, and it is the one I would publish.

A useful evaluation also includes untouched documents from the intended workflow, reviewed by someone qualified to establish the answers. I would keep those documents separate from prompt development. Synthetic sensitivity tests and representative task tests answer different questions; a product needs both.

Costs belong beside the score: image preparation, model requests, latency, human adjudication, and retries. The fastest model call can lose its advantage when a person has to repair every ambiguous chart.

I would publish the answer key and scoring rules with any eventual results. That is the concrete requirement behind verification literacy: another person can inspect what passed.

What to do Next

I would start with one chart from the intended workflow. The question and answer key come first. Then I would add a change that must reverse the answer, a change that must preserve it, and a case where uncertainty is correct.

I would run that small exercise through the full input path, including upload conversion and extraction. When a case failed, I would preserve the original input, transformed input, response, and adjudication. That is the evidence I would need to choose a repair.

I would decide what the product does with a failure. It might request a clearer image, expose the extracted values for confirmation, or hold the downstream action. The choice depends on the consequence, as in the agent permission model.

I would not connect chart interpretation to an Orbyt Collective action on the strength of a fluent explanation. I would first require the answer to survive changes that expose what it depends on. This article proposes that evaluation; it reports no new model run.

I would ask one question of every result: what changed in the evidence, and did the answer change for the right reason?

Related reading:

• Context Is the New Codebase. preserving the inputs that determine an agent's work.
• Verification Is the New Literacy making acceptance an explicit engineering responsibility.
• The Test Suite Became Orbyt's Only Living Specification. expressing the requirement in a test that can reject a plausible answer.
• Codex Accuses. Claude Convicts. why I test the evaluator as well as the proposed answer.

I trust a chart answer only as far as the evidence changes it can survive.