Your AI Prototype Works. Here Is Why It Will Fail in Production

A demo proves the model can do the job once, for you, on data you picked. It tells you nothing about customer 400, asking about customer 400's data, on the afternoon your model provider returns errors for twenty minutes.

Most AI features stall at the same point. Ten friendly accounts love the beta, and sales wants it on for everyone. Then three things land in the same month. An enterprise prospect asks in writing how customer data is isolated inside the AI feature. Finance asks what it costs per seat. A model update changes the answers, and a customer notices before you do.

This is the list we run before an AI feature leaves beta. Six gaps. None of them shows up in a demo, and each one comes with a test you can run this week.

Gap 1: Access. The model sees whatever retrieval hands it

The demo had one user and one set of documents. In production, retrieval is a new read path into your data, and it usually skips the authorization model the rest of the product relies on.

Here's the typical failure. Every tenant's embeddings sit in one index, and the tenant filter lives in application code that one route forgot. Or the tenant filter exists but the role check doesn't, so the most junior user in a workspace can ask the assistant about the board deck.

Treat retrieval like any other query that touches customer data:

• Tenant and permission filters go inside the retrieval query, not after it. Filtering afterwards hurts quality too: you fetch ten chunks, throw away eight and answer from two.
• The AI path calls the permission check your UI already uses. Don't rewrite it for the model.
• Tools the model calls run with the requesting user's permissions, never with a service account.

Then try to break it. Log in as a user in tenant A and ask for document names that exist only in tenant B. Repeat as the lowest role in tenant A. Any answer other than "I don't know" blocks the release.

Gap 2: Quality. "It seems better" is not a measurement

Nobody measured the demo. Someone watched it and nodded. In production you'll change the prompt, the chunking, the model and the corpus, and every change moves quality somewhere. Without evals, your customers find the regressions for you.

You don't need much to start:

• 50 to 200 real questions from real users, with answers approved by someone who knows the domain.
• A check per question: did the right source land in the top five, does the answer match, do the citations support the claim.
• A CI gate that fails the build when any metric drops by more than 5%.

A homegrown eval runner fits in under 300 lines. The tool matters far less than running it on every change.

If that sounds like a project, start with 20 questions your beta users actually asked, with the answers they should have gotten. Run them against the current version and the one before your last prompt change. Can't say which is better? Then you don't have evals yet.

Gap 3: Failure paths. Your provider will have a bad day

Calls fail in production. They time out, hit rate limits, return malformed JSON, or come back empty because retrieval found nothing. None of that happened in the demo, and each case needs a designed outcome, not a stack trace in the UI:

• a timeout on every model call, sized to the experience;
• retries with backoff and a cap, because every retry is billed;
• a fallback model or provider behind a gateway;
• an explicit "I don't know" path when retrieval returns nothing relevant.

Block the provider's hostname in staging and use the feature. Then force an empty retrieval result. Whatever the user sees is your production behavior today.

Gap 4: Cost. Price it per active customer, not per call

The demo cost a few dollars for the week, and that number predicts nothing. An illustrative calculation: take a mid-tier model at $3 per million input tokens and $15 per million output. A typical request carries 8,000 input tokens once you count the system prompt, retrieved chunks and chat history, and returns 500.

• One request: 8,000 × $3/M + 500 × $15/M, about $0.03.
• An active user at 20 requests a day: about $19 a month.
• Add 10% for retries: close to $21 per active user per month.

On a $29 seat, the feature just ate most of the margin on your most engaged customers. The cheapest levers are dull ones. Trim context: going from 8,000 to 3,000 input tokens cuts input cost by more than half. Cache repeated questions, which typically saves 30 to 50% of model spend. And route simple requests to a cheaper model.

Run your own version from last week's logs: average input tokens, output tokens and requests per active user, times your model's prices, next to what that customer pays you.

Gap 5: Visibility. You can't debug what you didn't record

A customer says the assistant was wrong yesterday. Now you need a trace of that request: user, tenant, prompt version, model version, retrieved chunk IDs, tokens, latency, cost, tool calls and the final output. If the feature can write anything, add an audit record as well. Who asked, what the model proposed, who approved it, what changed.

Try it on one odd answer from the beta and reconstruct exactly what the model saw. If that takes longer than ten minutes, you're not ready for a thousand users.

Gap 6: Rollout. Prompts and models are code

A prompt change is a deploy that changes behavior for every customer at once, so treat it like one. Prompts live in the repository and get reviewed like code. Model upgrades are releases, and they run the eval set first. The feature sits behind a per-tenant flag, starts at 10% of accounts, and has a kill switch that needs no deploy.

Ask how you'd roll back last Tuesday's prompt change. If the answer depends on someone remembering the old text, fix that first.

The shape of a production AI request

request
  -> auth context: user, tenant, role
  -> retrieval filtered by tenant and permissions      gap 1
  -> versioned prompt                                  gap 6
  -> model call via gateway: timeout, retry, fallback  gap 3
  -> output validation: schema, citations              gaps 2, 3
  -> action, with approval if it writes                gap 5
  -> trace and cost record                             gaps 4, 5
  -> sample into the eval set                          gap 2

Nothing here is exotic. It's the discipline you already apply to payments and authentication, pointed at a new read path and, sometimes, a new write path. I'd rather ship one AI feature that closes all six gaps than five that demo well.

The six-question check

Two or more "no" answers mean the feature is still a beta, whatever the label says.

1. Has someone tested whether a user in one tenant can get content from another tenant through the AI feature?
2. Do at least 50 real questions with approved answers run on every prompt or model change?
3. Do you know what the user sees when the provider is down, and when retrieval returns nothing?
4. Do you know the monthly cost per active customer, including retries and background jobs?
5. Can you reconstruct any answer from last week, including everything the model saw?
6. Can you turn the feature off for one tenant, or roll back a prompt, without a deploy?

What closing the gaps looks like

• Week 1: map every path from user to model and back, write the first 50-question eval set, run the tenant-leak test, build the cost model from real logs.
• Weeks 2 and 3: fix what the review found. That's usually retrieval filters first, then failure paths, tracing and the eval gate in CI.
• Weeks 4 and 5: roll out behind a per-tenant flag, 10% of accounts first.

We built Irresistible Bot for Vrinda Normand in one month for $5,000. The model was the easy part. What made it useful was structured onboarding that collects each user's brand context before the first generation, so every output is grounded in something real. Most AI features work the same way. What decides production is what context the model gets, from whom, and how you know the answer was good.

Related reading

• RAG done right in 2026: hybrid search, reranking, evals: the retrieval patterns behind gaps 1 and 2
• How Vrinda Normand scales her coaching business with an AI copywriting agent: the build above
• How we use Claude Code in production: how we build fast without handing the architecture to the tool

The service: AI Product Development. To bring in an engineer directly: hire an AI engineer.