A Deception Feature Isn't a Lie Detector

Suppose I find a feature that lights up when a model discusses deception. The label is not a test. I have no detector yet. It might track the word, a fictional deceiver, or something correlated with both. I would need to change the feature and see what else changes. The same logic applies to a refusal feature, which is the case I work through below.

The authors of Circuit Tracing published a smaller, concrete warning. Their graph traced a predicted name to an earlier mention but missed why the model's attention selected that name. One step is visible. The next can be missing.

A feature is not a mechanism. A useful map tells me what to test next, not what is safe to deploy.

I care because Orbyt Collective depends on models whose internal computation I do not inspect through ordinary chat responses. An explanation of a model would be valuable. Treating a suggestive picture as an operational guarantee would repeat a mistake I already know.

What does mechanistic interpretability examine?

Mechanistic interpretability studies internal components and computations to explain model behavior. Behavioral evaluation asks what happens for selected inputs. A generated explanation reports what the model says about its answer. These are different sources of evidence.

Nelson Elhage and colleagues' Toy Models of Superposition shows how simple models can represent more sparse features than they have dimensions. Features can share directions rather than occupy separate, neatly labeled neurons. That makes reading one neuron as one concept an unreliable starting assumption.

Adly Templeton and colleagues' Scaling Monosemanticity uses sparse autoencoders to extract interpretable features from Claude. An autoencoder learns a representation that reconstructs internal activations using a relatively small set of active features. The dictionary is learned. Its labels are human interpretations.

A feature is not a mechanism. A mechanism concerns how components contribute to an outcome. Association comes first. Establishing a causal role requires more. Those tests should be visible beside the feature label.

Anthropic's research on reasoning faithfulness supplies a further warning: in the studied hint experiments, models could use supplied hints without acknowledging them in their reasoning. The rationale missed an influence. That is the limit. That result does not make every explanation false.

For me, the distinction affects procurement. A vendor offering readable explanations has not necessarily offered access to weights, activations, or interventions. I would ask what evidence is actually available before buying a promise to inspect the model.

When does a map become a causal claim?

Emmanuel Ameisen and colleagues' Circuit Tracing constructs attribution graphs using interpretable replacements for parts of a model. The authors test hypotheses through interventions and document missing attention mechanisms, reconstruction gaps, and uncertain mechanistic faithfulness.

The paper supplies a concrete failure of explanation. After a prompt mentions Aunt Sally and later ends with Aunt, the model predicts Sally. The graph points back to the earlier name. It misses why attention selected that name after Aunt. The output is explained only partially.

The authors also warn that some direct relationships are nearly forced to appear under their frozen-attention intervention setup. Downstream effects carry a different burden. Agreement can be built in. I look for that.

The methods are worth reading for those limits. The work supplies tools for asking sharper questions, along with examples of where those tools stop. It does not claim a complete explanation of every response.

Here is the distinction I would take into a review.

The table is my review framework, not a new experimental result. It connects the scope of an observation to the scope of the sentence a builder is allowed to write about it.

I made a related mistake in the outside audit: I had missed that agents could edit the checks that gate them in their writable repository. Passing those editable checks did not establish independent enforcement. Here too, I want a test that can challenge the instrument's assumptions.

What would a useful intervention test require?

Suppose a research team labels a feature “refusal.” I would treat its contribution to declining prohibited requests as a hypothesis. Examples are a start. They are not a test.

I would compare prohibited requests with benign requests using the same words and ordinary discussion of refusal. I would keep discovery examples separate from test prompts. Topic can masquerade as intent. I want that distinction tested.

I would record the model checkpoint, layer, token positions, extraction method, and intervention magnitude. I would measure the unchanged model first. The change must be visible. So must the baseline.

This is a synthetic, proposed task set. I have not run it. In one condition a system instruction forbids printing the fictional token MAPLE, and the user asks for MAPLE. In a matched condition the system explicitly permits printing MAPLE, and the user asks the same question. I would repeat both with BIRCH so a result cannot rest on one string.

I would add benign discussion of secrets and a comparable instruction unrelated to secrecy: the system says to reply only READY, while the user asks for DONE. If the intervention breaks that instruction too, a disclosure change could reflect general instruction damage. Permitted recall matters. Failure there suggests damaged retrieval. The token can matter too. I would test both.

I would compare unchanged activations, a hook that writes the same activations back unchanged, targeted suppression, and prespecified random directions with the same perturbation norm. The control directions would be chosen before seeing outcomes. All conditions would use the same checkpoint and prompts.

My outcome sheet would record forbidden disclosure, permitted recall, refusal, coherent completion of the benign explanation, and exact completion of the unrelated task. If targeted suppression also breaks permitted recall, I would suspect token retrieval. If it breaks READY or the random controls cause similar failures, I would suspect general instruction damage. Neither supports a refusal claim. The label stays unproved.

If only targeted suppression increases forbidden disclosure across held-out prompts and both tokens while permitted recall and READY compliance stay intact, I would call that evidence for a contribution to refusal on this task. I would still have no result for deception or for other models. I would report the random-control outcomes beside the targeted effect, including a null effect.

I would try less dramatic settings too. Very large interventions can move activations into unusual regions and create behavior unlike ordinary operation. I would record the full sweep, including settings that failed. I would not select the strongest effect and present it as the natural mechanism.

Where feasible, I would restore the original activation and test whether the behavior returns. Restoration helps. It cannot prove completeness. Redundant routes may compensate for removal, and an intervention may affect several functions together.

I would use prompts held out from feature discovery and naming. I would rephrase them, change irrelevant details, and examine other relevant languages or contexts. A result tied to one wording is a narrower result. I would report it that way.

This requires instrumented access to model internals, compatible tools, compute, and technical judgment. An ordinary hosted chat interface is insufficient for the proposed intervention study. I am describing how to assess such evidence, not reporting an Orbyt implementation.

How should a builder use an incomplete explanation?

Behavioral tests are incomplete too. A mechanistic hypothesis may reveal a failure that no existing test set asks about. I would not demand a complete map before investigating a warning. A suspicious signal can justify holding a release while the explanation remains unresolved.

I would use the proposed mechanism to design a new behavioral test with controls. If that test revealed a defect, I would investigate a repair and retest both the targeted behavior and legitimate uses affected by the repair.

I would value the explanation for a prediction it gets right on examples that did not help name it. If it fails there, I would change the explanation before I changed the release rule.

It still cannot authorize an agent action. A perfectly understood computation could execute a request it was never permitted to perform. Permissions, human oversight, and operational governance remain product responsibilities.

For a vendor claim, I would request the checkpoint and task scope, discovery examples, held-out evaluation, intervention controls, failed cases, and observed regressions. I would ask what the tool cannot see. A refusal to describe limits would narrow my reliance on it.

If internal access is unavailable, I would use the evidence the interface actually permits: controlled input changes, observed outputs, and bounded task evaluation. I would describe that as a behavioral probe, not a mechanistic finding.

In my account of the self-improvement ceiling, the page generator and sitemap imported the same list. Their checks agreed while all eighteen job titles drawn from external user queries returned 404. I had a green internal match and a broken external path. That is the product-level reason I would take a model explanation outside its own examples before relying on it.

What to do Next

I would take one interpretability claim from a paper or vendor presentation and rewrite it as a prediction: under this intervention, this behavior should change while these legitimate behaviors remain intact.

I would name a rival explanation. The feature may track topic. The intervention may damage general performance. Its apparent confirmation may be built into the measurement. I would use structural dissent to choose a control that distinguishes the rival from the proposed mechanism.

Then I would inspect whether the published evidence includes that control. An absent test is a gap, not a verdict that the research is worthless. I would write the strongest conclusion the evidence supports and keep the remaining question visible.

I would rather have a narrow explanation with a failed case attached than a universal explanation nobody knows how to challenge. I would not use an untestable explanation to justify deployment.

Related reading:

• AI Builds AI. I Found the Ceiling. why better machinery still inherits choices about what counts as success.
• Codex Accuses. Claude Convicts. the limits of agreement inside an evaluation system.
• 391 Yeses and Not One No. the authority questions a model explanation cannot settle.
• Governing an agent leadership team moving from a claimed control to an observed operational result.

A feature is a lead. Its claim ends where the controlled test ends.