A tilted glass panel of text lines with green and red highlights stands in black space, hit by green, white and red light beams, beneath a curved planet edge.

What a Synthetic Persona Cannot Tell You

A name on a prompt is not a witness.

Key findings

  • A persona used as a stand-in for people and a persona used as a critic fail in different ways.
  • An average that matches real people can hide answers that fail in the detail.
  • Several personas on one model count as one opinion until something independent checks them.
  • Keep a persona's point only when it still stands with the name taken off.

Key measurements

Each measurement, its value, and where and when it was measured.
MeasureValueSource and date
Regression coefficients from persona answers that differed significantly from human survey data48 percentMay 17, 2024 · Bisbee et al., Synthetic Replacements for Human Survey Data? (Political Analysis, 2024)
Language models in the correlated errors evaluationover 350Jun 9, 2025 · Kim et al., Correlated Errors in Large Language Models (ICML 2025)
Lenses on the Orbyt Labs roster39Nov 3, 2026 · Orbyt Labs, Synthetic Personas
Verdicts on record from a different model maker5 dissent, 0 concurNov 5, 2026 · Orbyt Labs, Synthetic Personas

A synthetic persona can tell you what to check. It cannot tell you what is true.

I keep a roster of 39 of them, counted on 2026-11-03. I have the records of what they did, and I have read what outside researchers found when they tested the idea. Both say a persona hands you a question, never an answer.

A synthetic persona produces critique. Read it as evidence and you are counting your own model's opinion as a witness.

This is the short version for anyone deciding whether to use one. The setup, the counts and the gaps in mine are on the Synthetic Personas page.

What are you asking the persona to do?

Two different jobs travel under one name.

One job is a stand-in. You ask a model to answer as a kind of person, and you read the answers as what those people think. Researchers call these silicon samples. You will also see them called synthetic users.

The other job is a critic. You give a model a perspective and point it at a draft. You read the answer as a question somebody should go check. That is how I use them, and I call them lenses.

The two fail differently. A wrong critic can file a bug that is not there and keep filing it. A wrong stand-in can replace the customer interview you never run.

Can a synthetic persona replace real respondents?

Not on the studies I read.

The best case first. Argyle and colleagues (opens in a new tab) conditioned GPT-3 on backstories from real participants in large US surveys. They argued a model could emulate the response patterns of many subgroups, and they called that property algorithmic fidelity.

Bisbee and colleagues (opens in a new tab) then tested the idea against people. They gave ChatGPT 3.5 Turbo personas built from American National Election Study respondents. The averages tracked the human answers closely. The synthetic answers varied less than people do. Of the regression coefficients, 48 percent differed significantly from the human data, and the sign flipped on 32 percent of those. Running the same prompt again after a model update did not reproduce the first result.

The average looked right. That is the trap.

Santurkar and colleagues (opens in a new tab) compared model opinions with 60 US demographic groups and found substantial misalignment. It persisted when they steered the model toward a group.

The strongest counterexample is worth reading closely. Park and colleagues (opens in a new tab) built agents from two hour interviews, surveys or both, using 1,052 Americans. Those agents predicted held out survey, personality and economic game responses. Accuracy gaps across racial and ideological groups shrank compared with agents given demographics alone.

Look at what those agents had. Each one was grounded in what a real person said.

That is the point of the review of 63 persona studies (opens in a new tab) presented at AIES. Only 35 percent discussed how representative their personas were, and its checklist asks for explicit grounding in empirical data.

Now look at mine. My prompts carry a name and one question. They do not carry the person's work. So a lens answers from what the model learned in training, and it can misstate the person it is named for. A lens of mine cannot stand in for anyone, and I do not ask it to.

Why does a panel of personas sound like one voice?

Often it is one voice.

Kim and colleagues (opens in a new tab) evaluated over 350 language models and found substantial error correlation. Sharing an architecture or a provider raised it. Larger, more accurate models had highly correlated errors even when their makers differed.

A persona panel is the tight version of that. In my seats a cabinet is applied in one run, on one model, to a draft the same run wrote. Four names do not make four readers.

My manuscript cabinet once voted four to none on a chapter. Every lens volunteered the same doubt about its own verdict, in the same direction. That is softer than the tally looks, and the book log says so. The rule I keep for this cuts that cabinet to two lenses after three unanimous passes in a row. A split would not prove independence either. It only proves the outputs differed.

Asking a different maker helps, and it has its own limit. My outside checker has five verdicts on record, counted on 2026-11-05. All five dissent. None concurs. So I cannot show that it would ever call anything sound.

Will a persona tell you that you are wrong?

Sometimes it tells you what you want to hear. Sometimes it invents a problem.

Sharma and colleagues (opens in a new tab) found five leading assistants matching users' views across four tasks, with evidence the habit comes in part from human preference judgments. Perez and colleagues (opens in a new tab) found larger models more likely to echo the answer a user seems to prefer. My reading, which neither study tests: state your opinion first and you have tilted the panel before it starts.

Panickssery, Bowman and Feng (opens in a new tab) found model evaluators rating their own output above other text that people judged equal. That is the setting when a seat holds its own draft against its own lenses.

The opposite error is real too. McAleese and colleagues (opens in a new tab) found model critics catching real bugs and sometimes reporting bugs that were not there. Teams of people and critics caught about as many, with fewer invented ones than the models alone.

So how often is a lens of mine wrong? I do not know. When the first evaluation of my lenses came due, I had not scored a single finding. I still have not.

What can a synthetic persona tell you?

Where to look.

Here is the test I use. Take the persona's name off the sentence and see what is left. If it still points at something you can check, keep it. If only the name gave it weight, drop it.

One of my seats drafted an idea: a job seeker's agent watching their inbox and pricing offers. Its self review applied a risk pattern from the work behind one lens: private data, untrusted content and an action path, together in one agent. You can examine that sentence with no persona in the room.

In the chapter vote, a lens whose question is where the demo is counted about eleven showable scenes on one side and one on the other. I can count those too.

My legal seat's cabinet carries no personas. It checks its work against named legal frameworks, because a legal cabinet of famous names is, in the board document's words, authority laundering at its most dangerous. That phrase covers the whole failure. A name is doing the work a source should do.

What a persona writes is generated text, and I never print it as that person's view. The board's rule keeps lens names internal, and each exception takes a recorded decision and the roster page's terms.

What to do Next

Before you trust a persona's answer, sort the question.

The questionWhat a persona can give youWhat answers it
What do customers think?Questions for an interviewCustomers
What would this person say?A guess that can misstate themTheir published work, or asking them
Is this draft correct?A place to lookA test, a source or a count
Are we all wrong?One argued caseA reader on another model, plus evidence
Did review improve the decision?NothingA scored record

Then do four things. Pick the job first, because a critic and a stand-in need different proof. Ground a stand-in in what real people said, since a name alone grounds nothing. Keep your own opinion out of the prompt. And score the findings as right, wrong or unchecked. I have not done that last one, and it is the reason I can only make the claims above.

None of this shows that lenses improve decisions. I have not measured that.

Related reading:

A name on a prompt is not a witness.

Methodology

Every Orbyt figure comes from the Synthetic Personas page and its generated facts, with the date each was counted. The manuscript vote comes from the book log. For the outside studies I read each abstract. For the Bisbee study I also read the article page, which is where the 48 and 32 percent figures appear.

Limitations

I did not run a persona study of my own. Most of the outside studies test personas as stand-ins for survey respondents, not as named critics, so they bear on my lenses only by analogy. No lens finding of mine is scored. Nothing here measures whether lenses improve decisions.

Sources

  1. Orbyt Labs, Synthetic Personas Retrieved Nov 5, 2026.
  2. Bisbee et al., Synthetic Replacements for Human Survey Data? (Political Analysis, 2024) (opens in a new tab) Retrieved Nov 5, 2026.
  3. Kim et al., Correlated Errors in Large Language Models (ICML 2025) (opens in a new tab) Retrieved Nov 5, 2026.
  4. Argyle et al., Out of One, Many: Using Language Models to Simulate Human Samples (opens in a new tab) Retrieved Nov 5, 2026.
  5. Santurkar et al., Whose Opinions Do Language Models Reflect? (opens in a new tab) Retrieved Nov 5, 2026.
  6. Park et al., LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals (opens in a new tab) Retrieved Nov 5, 2026.
  7. Batzner et al., Whose Personae? Synthetic Persona Experiments in LLM Research and Pathways to Transparency (opens in a new tab) Retrieved Nov 5, 2026.
  8. Sharma et al., Towards Understanding Sycophancy in Language Models (opens in a new tab) Retrieved Nov 5, 2026.
  9. Perez et al., Discovering Language Model Behaviors with Model-Written Evaluations (opens in a new tab) Retrieved Nov 5, 2026.
  10. Panickssery, Bowman and Feng, LLM Evaluators Recognize and Favor Their Own Generations (opens in a new tab) Retrieved Nov 5, 2026.
  11. McAleese et al., LLM Critics Help Catch LLM Bugs (opens in a new tab) Retrieved Nov 5, 2026.

Synthetic persona reliability questions? Answers.

Not on the studies I read. Averages can look right while the detail fails. One study built personas from election survey respondents. The averages tracked people, the answers varied less, and 48 percent of regression coefficients differed significantly from the human data. Use personas to write better questions, then ask real people.

Often it is one model answering under several names. In my seats a cabinet is applied in one run, on one model, to a draft the same run wrote. Outside research found errors correlate across language models, even larger ones from different makers. Treat a unanimous panel as one opinion until something independent checks it.

Finding the question. Take the persona's name off the sentence and see what is left. If it still points at something you can check, such as a count, a file or a source, keep it. If only the name gave it weight, drop it. The check answers the question, never the persona.

No. My prompts carry a name and one question, not the person's work, so the model answers from what it learned in training and can misstate them. Its output is generated text. I never print it as that person's view, and the board's rule keeps lens names internal, with narrow exceptions.

Justin Bartak

Founder, CEO and Chief AI Officer of Orbyt Labs

Builds software with AI agents across engineering, product, design and go-to-market. Publishes tests, failures and decisions at Orbyt Labs.

Writes The Machine Speaks with the agents that build the product, and The AI-Native Lens (opens in a new tab).

All of The Machine Speaks