The First Ten Minutes of an AI Product

If I have to explain which file an AI product accepts, I am part of its interface. A generated answer after that rescue is an assisted result. I would write down my explanation and test whether the product can supply it for the next newcomer.

The rescue changes the score.

I have not run this study. The supplier task below is synthetic. Its people, documents, dates, and decisions are invented so the proposed test can be scored, not mistaken for an Orbyt customer session.

The first useful result is the moment a person can judge the work without borrowing the founder's understanding. An account and a stream of text do not establish that moment.

What would count as a useful result?

I would write the acceptance condition before designing a welcome screen. In this synthetic example, the person needs to choose a supplier for 100 replacement parts. The stated requirement is delivery by May 1, 2027, with an insurance certificate valid on the delivery date. The product receives two invented supplier packets. Both are text PDFs.

The source pages matter.

I would call the initial task complete only if the person can open the cited pages, see Alder as the sole supplier supported by both requirements, see Birch's certificate as unknown, and explain why Birch fails the original deadline. No founder explanation counts toward independent completion.

When I change the deadline to May 5, I expect the comparison to update Birch's delivery status while keeping its certificate unknown. The person must be able to correct an extracted date and see the judgment update without starting over.

Those are observable conditions. A fluent paragraph that declares both suppliers qualified would fail. So would a correct answer whose source pages the person cannot inspect. I would score each condition separately. Stopped attempts count too.

This is the testable form of my design philosophy: the person should leave able to act, not merely impressed by an answer. It also connects to an admission I published in my audit of the audit loop: "I still did not apply it to the machinery itself." I had written an audit dimension that reads live production state and had not pointed it at the governance machinery itself. I would not let an onboarding checklist make the same mistake.

The failure in AI Builds AI. I Found the Ceiling. is specific. The page generator and sitemap imported the same list, so they agreed; eighteen job titles from real user queries returned 404. Agreement was the trap. An onboarding flow can likewise agree with its own checklist while missing the decision the person actually came to make. I would ask for that decision in the person's words before scoring a generated answer against my specimen.

Where would I start the clock?

I would start the clock at the agreed entry point. I would record what the person already knows and which materials they bring. A returning expert opening a prepared workspace is a different study from a newcomer following a product link.

For this proposed protocol, ten minutes is an observation window. It is not an industry benchmark or a universal product deadline. Ten minutes proves nothing alone. If the task needs longer, I would record that and choose a more appropriate window for the next study.

I would keep total elapsed time and record where it went: setup, searching for the next step, system waiting, and result inspection. A fast model cannot make an unexplained permission request disappear.

Here is the worksheet I would use. Its observation column would stay blank until a session happens.

I would include people who stop. Excluding incomplete sessions would erase the friction the study exists to find.

They are part of the result.

What would I stop doing as founder?

I cannot supply the missing interface and then count the result as independent. I would not point at the right control, explain an ambiguous label, or prepare a perfect input without recording that help.

When help becomes necessary, I would give it. I would record the trigger and my exact words. The person could continue, but I would report assisted completion and leave the independent attempt incomplete.

That separation lets the person continue without letting me erase the rescue from the evidence.

I would keep my sentence.

In the synthetic task, suppose the person tries to upload a scan. If the prototype accepts only text documents, I might have to say, "This version cannot read a scan. Please use a text PDF." That is an invented rescue sentence, not a line from an observed session. I would put the restriction before upload and offer a way to check the file, or I would add scan support. The first choice narrows the task the product can serve. The second adds work to the product. A welcome animation resolves neither choice.

I would build a small route. I made the case for prototypes over mood boards elsewhere. Here the prototype would answer one exact question: can the person reach and correct the supplier decision without my sentence?

What would permission need to explain?

I would ask permission when the reason is visible. Reading the supplied documents serves this task. Sending messages or inspecting unrelated workspace files would need a separate purpose. That scope must be clear. I would make the effect of declining clear before the person commits to setup.

Google's mental model guidance (opens in a new tab) recommends explaining an AI system's capabilities and limits. I would apply that guidance to the file restriction before someone prepares a scan the product cannot use. I would label sample data as sample data and let the person replace it. A polished sample would not count as evidence that their own documents work.

The principle behind invisible UX is useful here. I would remove unnecessary work while preserving the decisions a person must understand: what I read, what I cannot read, and what I will do with it.

Could the clock reward a shallow product?

Yes. A quick demonstration can meet an easy condition while failing the person's actual task. The synthetic specimen makes the scoring rule explicit. It cannot choose the real task. Before a real session, I would ask the person what decision they need to make, what evidence they trust, and what a wrong answer would cost them. If the supplier exercise does not match that work, I would replace the exercise rather than announce success.

For a complex installation, I might measure a verified connection and a clear preview as a setup milestone. I would label it setup. Setup is not the task. I would still track whether the person later completes the research task that justified the installation.

I would keep setup completion, first task completion, assisted completion, and return use separate. I would test whether the person can reopen the work and continue after the introduction. None of those observations alone establishes retention, recurring value, willingness to pay, or customer demand. Orbyt has no paying customers to cite here.

I would report the task, relevant experience, environment, completed and incomplete attempts, and interventions. A small study could expose a broken route. It could not produce a market-wide completion rate from a few attempts.

What to do Next

I would choose one intended user and one real task. I would write the acceptance condition, entry point, allowed assistance, and observation window. I would prepare safe material and get permission to record the session.

I would watch without narrating the product. If I had to help, I would keep my sentence beside the result. I would fix the first obstruction. Then I would test again. I would ask a fresh participant to try the revised route and preserve the earlier attempt for comparison. I have not run this exercise, so I have no before and after result to report.

Related reading:

• Simple Is Expensive. on the decisions a team absorbs to make the first experience clear.
• Design Is the Fastest Way to Build Trust on setting expectations before the result appears.
• The Test Suite Became Orbyt's Only Living Specification. on defining useful behavior before claiming success.
• Building AI-Native Apps with Taste on matching capability to the experience of using it.

The founder must stop supplying the missing interface.