<!-- orbyt:chrome:begin -->
---
title: "What a Synthetic Persona Cannot Tell You"
description: "A synthetic persona can hand you a question worth checking. It cannot tell you what a person thinks, what customers want, or whether a draft is right. Published studies find persona answers flatten real variation and models share errors, so a panel can be one voice. Use personas to find questions, then answer them with evidence."
author: "Justin Bartak"
published: 2026-11-20
modified: 2026-11-20
canonical: https://www.orbytlabs.ai/blog/what-a-synthetic-persona-cannot-tell-you
---
<!-- orbyt:chrome:end -->

# What a Synthetic Persona Cannot Tell You

A synthetic persona can tell you what to check. It cannot tell you what is true.

I keep a roster of 39 of them, counted on 2026-11-03. I have the records of what they did, and I have read what outside researchers found when they tested the idea. Both say a persona hands you a question, never an answer.

**A synthetic persona produces critique. Read it as evidence and you are counting your own model's opinion as a witness.**

This is the short version for anyone deciding whether to use one. The setup, the counts and the gaps in mine are on the [Synthetic Personas](https://www.orbytlabs.ai/research/synthetic-personas) page.

## What are you asking the persona to do?

Two different jobs travel under one name.

One job is a stand-in. You ask a model to answer as a kind of person, and you read the answers as what those people think. Researchers call these silicon samples. You will also see them called synthetic users.

The other job is a critic. You give a model a perspective and point it at a draft. You read the answer as a question somebody should go check. That is how I use them, and I call them [lenses](https://www.orbytlabs.ai/orbyt-collective/glossary/lenses).

The two fail differently. A wrong critic can file a bug that is not there and keep filing it. A wrong stand-in can replace the customer interview you never run.

## Can a synthetic persona replace real respondents?

Not on the studies I read.

The best case first. [Argyle and colleagues](https://arxiv.org/abs/2209.06899) conditioned GPT-3 on backstories from real participants in large US surveys. They argued a model could emulate the response patterns of many subgroups, and they called that property algorithmic fidelity.

[Bisbee and colleagues](https://www.cambridge.org/core/product/B92267DC26195C7F36E63EA04A47D2FE) then tested the idea against people. They gave ChatGPT 3.5 Turbo personas built from American National Election Study respondents. The averages tracked the human answers closely. The synthetic answers varied less than people do. Of the regression coefficients, 48 percent differed significantly from the human data, and the sign flipped on 32 percent of those. Running the same prompt again after a model update did not reproduce the first result.

The average looked right. That is the trap.

[Santurkar and colleagues](https://arxiv.org/abs/2303.17548) compared model opinions with 60 US demographic groups and found substantial misalignment. It persisted when they steered the model toward a group.

The strongest counterexample is worth reading closely. [Park and colleagues](https://arxiv.org/abs/2411.10109) built agents from two hour interviews, surveys or both, using 1,052 Americans. Those agents predicted held out survey, personality and economic game responses. Accuracy gaps across racial and ideological groups shrank compared with agents given demographics alone.

Look at what those agents had. Each one was grounded in what a real person said.

That is the point of the [review of 63 persona studies](https://arxiv.org/abs/2512.00461) presented at AIES. Only 35 percent discussed how representative their personas were, and its checklist asks for explicit grounding in empirical data.

Now look at mine. My prompts carry a name and one question. They do not carry the person's work. So a lens answers from what the model learned in training, and it can misstate the person it is named for. A lens of mine cannot stand in for anyone, and I do not ask it to.

## Why does a panel of personas sound like one voice?

Often it is one voice.

[Kim and colleagues](https://arxiv.org/abs/2506.07962) evaluated over 350 language models and found substantial error correlation. Sharing an architecture or a provider raised it. Larger, more accurate models had highly correlated errors even when their makers differed.

A persona panel is the tight version of that. In my seats a cabinet is applied in one run, on one model, to a draft the same run wrote. Four names do not make four readers.

My manuscript cabinet once voted four to none on a chapter. Every lens volunteered the same doubt about its own verdict, in the same direction. That is softer than the tally looks, and the book log says so. The rule I keep for this cuts that cabinet to two lenses after three unanimous passes in a row. A split would not prove independence either. It only proves the outputs differed.

Asking a different maker helps, and it has its own limit. My outside checker has five verdicts on record, counted on 2026-11-05. All five dissent. None concurs. So I cannot show that it would ever call anything sound.

## Will a persona tell you that you are wrong?

Sometimes it tells you what you want to hear. Sometimes it invents a problem.

[Sharma and colleagues](https://arxiv.org/abs/2310.13548) found five leading assistants matching users' views across four tasks, with evidence the habit comes in part from human preference judgments. [Perez and colleagues](https://arxiv.org/abs/2212.09251) found larger models more likely to echo the answer a user seems to prefer. My reading, which neither study tests: state your opinion first and you have tilted the panel before it starts.

[Panickssery, Bowman and Feng](https://arxiv.org/abs/2404.13076) found model evaluators rating their own output above other text that people judged equal. That is the setting when a seat holds its own draft against its own lenses.

The opposite error is real too. [McAleese and colleagues](https://arxiv.org/abs/2407.00215) found model critics catching real bugs and sometimes reporting bugs that were not there. Teams of people and critics caught about as many, with fewer invented ones than the models alone.

So how often is a lens of mine wrong? I do not know. When the first evaluation of my lenses came due, I had not scored a single finding. I still have not.

## What can a synthetic persona tell you?

Where to look.

Here is the test I use. Take the persona's name off the sentence and see what is left. If it still points at something you can check, keep it. If only the name gave it weight, drop it.

One of my seats drafted an idea: a job seeker's agent watching their inbox and pricing offers. Its self review applied a risk pattern from the work behind one lens: private data, untrusted content and an action path, together in one agent. You can examine that sentence with no persona in the room.

In the chapter vote, a lens whose question is where the demo is counted about eleven showable scenes on one side and one on the other. I can count those too.

My legal [seat's](https://www.orbytlabs.ai/orbyt-collective/leadership) cabinet carries no personas. It checks its work against named legal frameworks, because a legal cabinet of famous names is, in the board document's words, authority laundering at its most dangerous. That phrase covers the whole failure. A name is doing the work a source should do.

What a persona writes is generated text, and I never print it as that person's view. The board's rule keeps lens names internal, and each exception takes a recorded decision and the [roster page's](https://www.orbytlabs.ai/orbyt-collective/influences) terms.

## What to do Next

Before you trust a persona's answer, sort the question.

| The question | What a persona can give you | What answers it |
| --- | --- | --- |
| What do customers think? | Questions for an interview | Customers |
| What would this person say? | A guess that can misstate them | Their published work, or asking them |
| Is this draft correct? | A place to look | A test, a source or a count |
| Are we all wrong? | One argued case | A reader on another model, plus evidence |
| Did review improve the decision? | Nothing | A scored record |

Then do four things. Pick the job first, because a critic and a stand-in need different proof. Ground a stand-in in what real people said, since a name alone grounds nothing. Keep your own opinion out of the prompt. And score the findings as right, wrong or unchecked. I have not done that last one, and it is the reason I can only make the claims above.

None of this shows that lenses improve decisions. I have not measured that.

Related reading:

- [The Tenth Man Is on Payroll.](https://www.orbytlabs.ai/blog/red-teams-tenth-man) the case for assigned dissent, and why correlated agreement is not independent agreement.
- [Codex Accuses. Claude Convicts.](https://www.orbytlabs.ai/blog/two-ai-audit-loop-codex-claude) what a reviewer from another lab found that my own checks could not.
- [Verification Is the New Literacy](https://www.orbytlabs.ai/blog/verification-is-the-new-literacy) why an observation has to stand apart from the story it supports.

**A name on a prompt is not a witness.**

<!-- orbyt:chrome:begin -->
This is the markdown version of https://www.orbytlabs.ai/blog/what-a-synthetic-persona-cannot-tell-you. The page there is the canonical copy.
<!-- orbyt:chrome:end -->
