A Fixed-Price AI Project Needs a Narrow Acceptance Test

So the vendor bills by the hour. Every review meeting surfaces a new input that comes out wrong, and it becomes a ticket. Nobody can say when the work is done, because nobody wrote down what done means. Six months in, the founder is paying for a project that never failed and never finished.

The first half of that sentence is true. A model phrases the same answer differently from one day to the next, and some inputs it'll get wrong forever. The second half doesn't follow. The output varies. The test you judge it by doesn't have to.

We sell AI work at a flat price, and our service page says the uncomfortable part out loud: if we underestimate, we eat it. That only works because the finish line is written down before the first line of code, as a narrow acceptance test both sides sign.

If you're weighing an AI quote, or trying to get one that isn't "time and materials, we'll see", this is the document to ask for. It has four parts.

Why AI projects drift

Without a fixed set of inputs, every review is a new test. Someone tries a question they thought of in the shower. It comes out wrong, and a ticket appears. Fix that one and the next review finds another. The work isn't bad. It just has no finish line.

Each of the four parts below closes one way a project drifts. Skip one and the drift comes back through that gap.

Part 1: a frozen test set

Collect real inputs before the build starts: real emails or support questions, whatever the feature will see in production. Somewhere between 50 and 200 is usually enough. It's the same range we use for evals on retrieval systems. The buyer supplies them, because the buyer knows what real looks like and we don't yet.

Then freeze it. Inputs anyone thinks of halfway through go into the set for version two, not into this contract. That one rule removes most of the arguments before they start.

Part 2: a pass bar for each check

For every input, define what a pass is. Some checks are mechanical, and code can run them: the output is valid JSON with the required fields, and it never quotes a price that isn't in the source. Others need a person. A reviewer from the buyer's side marks each output as usable or not.

Then turn the bar into numbers. An illustrative version:

• at least 45 of 50 drafts rated usable without a rewrite;
• zero outputs that invent a fact about the customer;
• every response back in under ten seconds;
• model cost per request under an agreed cap at the expected volume.

The exact numbers are a negotiation. Having numbers at all is the point.

Two rules keep this honest. The reviewer is named before the build starts, so "usable" means one person's judgment and not a committee's mood. And the bar sits below perfect on purpose. If a model scores 50 out of 50, the test set was probably too easy.

Part 3: fallback behavior

A test set measures the happy path. Production also has the provider timing out and the question the data can't answer. Those need a defined outcome, and the outcome belongs in the acceptance test.

"When the source has no answer, the assistant says so and offers a person" is testable. "Handles edge cases gracefully" isn't. Put a handful of these failure cases into the test set on purpose, next to the normal inputs. They're also where much of the production work goes, which is why they're gap 3 in our AI production checklist.

Part 4: the out-of-scope list

This is where a fixed price actually lives. Write down what version one doesn't do: other languages, other channels, volume above a set number, integrations beyond the one named, the admin screen someone will ask for in week three. Every line on that list is a conversation you won't have to have in the middle of the project.

Who carries which risk

A flat price moves the estimation risk to us. If the work takes longer than we thought, that's our cost, not yours. The acceptance test moves the scope risk into the open. If the goal changes, both sides can see it change, and it becomes a new, separately priced piece of work instead of a slow leak in the current one.

Neither side gets everything. The buyer gives up the right to add inputs mid-build. We give up the right to bill for our own misjudgment. I think that's a fair trade, and I haven't found a better one.

What narrow looks like

Irresistible Bot is a good picture of a narrow job. It's an AI copywriting agent in Vrinda Normand's voice. It writes email sequences, social posts and landing page copy, and it guides her students through her copywriting frameworks. Structured onboarding collects each user's brand context before the first generation. We built it in one month for a fixed $5,000.

The whole job fits in a few sentences, and that's what makes a fixed number possible. You can put a narrow job in front of a reviewer and ask whether each draft is usable. You can't do that with "an AI assistant for my business."

What we won't fix-price

Some projects can't have an acceptance test yet, and we'd rather say so than pretend.

• Open research, as in "see what AI can do with our data."
• A goal nobody can measure, like "make it smarter."
• No real inputs to test with, because the workflow doesn't exist yet.
• A spec that changes every week. Our MVP pricing post names the same exception.

In those cases the first deliverable should be the test set itself. Once it exists, the rest can be priced.

A one-page acceptance test

Job           what the feature does and for whom, in two sentences
Test set      N real inputs, supplied by the buyer, frozen on [date]
Checks        mechanical (run by code) and human (named reviewer)
Pass bar      a number per check, e.g. 45 of 50 usable, 0 invented facts
Fallbacks     no answer, provider error, unplanned language, off-topic input
Out of scope  languages, channels, volume, integrations, extra screens
Sign-off      who runs the test and what happens on a fail

Before you sign

1. Is there a test set of real inputs, and do you know who supplied it?
2. Is it frozen, with a rule for inputs that come up later?
3. Does every check have a number instead of an adjective?
4. Is the reviewer named?
5. Are failure cases in the test set, not just the happy path?
6. Is there a written out-of-scope list?
7. Does the contract say what happens if the test fails?

Any "no" here is a place where the price can still move after you sign.

Related reading

• MVP cost in 2026: real numbers from $5K to $50K: why a flat price assumes stable scope
• Hire vs engage: the 2026 math with AI agents: the numbers behind a scoped engagement versus a hire
• Why we don't take equity: why we price in cash, and what that does to incentives

The service: AI Product Development. How flat pricing works: flat-price engagement. Comparing vendors: agency vs Start Matter.