AI UAT sample size: zero failures in 20 runs isn't a pass, so here's how many runs you need
Picture the UAT wrap-up for an AI refund agent. The test lead puts up one slide: "20 test runs, 0 failures." The product owner smiles. Somebody says "ship it." And nobody in the room asks the question that matters, which is: how bad could this agent be and still have produced that slide?
The answer is about 14%. An agent that gets one case in seven wrong will still hand you a clean 20 out of 20 about one time in twenty, so that slide can't rule it out at 95% confidence. Your slide is true. It just doesn't say what everyone in the room thinks it says.
I'm a business analyst, and this is the part of AI testing I think BAs should own, because it isn't really a testing question. It's a requirements question. "How many runs do we need?" depends entirely on "how wrong are we allowed to be, and for which kind of case?", and only the business can answer that. So in this post I'll show the math in plain words, a 30-line Python script you can run with nothing installed, a filled sign-off table on a fictional Northstar Systems agent, and what I'd change in a UAT plan on Monday.
Why this matters more for AI than for ordinary software
For a normal feature, UAT is mostly about coverage. If the refund screen calculates tax correctly for an order in North Carolina, it'll do it the same way tomorrow. One good run per scenario is evidence, because the code is deterministic.
An AI feature isn't like that. The same input can come back with a different answer on a second run, so a passing case is a sample, not a proof. You're no longer asking "does it work?" You're asking "how often does it fail?", and that's a rate. Rates need counts.
This isn't a hunch. When Shunyu Yao and colleagues built τ-bench, a benchmark where agents talk to simulated customers and call tools in retail and airline settings, they added a metric called pass^k: the chance that an agent gets the same task right on all k tries. Their abstract reports that even strong function-calling agents "succeed on <50% of the tasks," and that pass^8 fell below 25% in the retail domain. An agent that looks fine once can look very different on the eighth try.
I've written before that an agent doesn't pass a test, it passes a pass rate. That template tells you to set the pass mark per bucket before the first run and to run the golden set three times. Three runs is a good gate for spotting instability. It isn't enough to put a number on it. This post is about the number.
The V-model pairs each requirement level with the test that proves it. With AI features, the acceptance test on the right side needs a sample size, and that number comes from the requirement on the left. Image: "V-model.svg" by Herman Bruyninckx, CC BY-SA 3.0, via Wikimedia Commons.
The rule of three, in plain words
Here's the one fact worth memorizing. If you ran n trials and saw zero failures, the true failure rate could still be as high as about 3 divided by n, at 95% confidence.
That's the rule of three. The Wikipedia page puts it as "the interval from 0 to 3/n is a 95% confidence interval for the rate of occurrences." It comes from a simple question. If the real failure rate is p, the chance of n clean runs in a row is (1 − p) to the power n. Set that equal to 5% and solve. The natural log of 0.05 is about −2.996, which rounds to −3, and you land on p ≈ 3/n.
So:
| Clean runs (0 failures) | Failure rate you can't rule out (rule of three) | Exact figure |
|---|---|---|
| 10 | 30% | 25.9% |
| 20 | 15% | 13.9% |
| 30 | 10% | 9.5% |
| 59 | 5.1% | 5.0% |
| 100 | 3% | 3.0% |
The "exact" column comes from the script further down. The page notes the rule is a good approximation once n is above 30, and the table shows why: at 10 runs it overstates the bound a bit, at 100 it's spot on. Either way, the shape is what matters. Every time you want to halve the failure rate you can rule out, you need about twice as many clean runs.
Turn that around and you get the planning version: to claim a failure rate under X% with zero failures, you need about 300 divided by X runs.
- Under 10%: about 30 runs (exact: 29)
- Under 5%: about 60 runs (exact: 59)
- Under 2%: about 150 runs (exact: 149)
- Under 1%: about 300 runs (exact: 299)
I didn't come up with the 59 and the 149. A reader called aiapikey made exactly this point in the comments under Michał Woliński's post, How to Evaluate AI Agents: Grade the World, Not the Transcript: that 0 of 20 still allows a false-pass rate up to about 14%, and that claiming under 5% takes about 59 clean trials, and under 2% about 149. I reran both numbers in the script below and got the same answers. It's the kind of comment that should change how a whole team writes its test plan, so it's worth crediting by name.
What if you saw a failure?
Zero failures is the easy case. Real UAT has a failure or two, and then the rule of three no longer applies. You need the exact version, usually called the Clopper-Pearson interval. Wikipedia calls it "an early and very common method" and notes it's often called "exact" because its coverage "is never less than the nominal." In plain words, it's cautious. It won't tell you things are better than they are.
You don't need to know the formula. You need to know how fast one failure moves the bound:
| What you saw | Failure rate you can't rule out (95%, exact) |
|---|---|
| 0 of 20 | 13.9% |
| 1 of 20 | 21.6% |
| 1 of 50 | 9.1% |
| 2 of 50 | 12.1% |
| 1 of 100 | 4.7% |
| 3 of 200 | 3.8% |
Look at the second row. One failure in 20 runs, which a lot of people would read as "95% pass rate, fine," still leaves room for an agent that fails more than one case in five. The point estimate is 5%. The honest statement is "somewhere between almost zero and 21.6%."
That gap between the point estimate and the bound is where most AI sign-off slides go wrong. The slide reports the point estimate. The risk lives in the bound.
The script: 30 lines, standard library only
Here's the artifact for this post. It computes the exact one-sided upper bound for any count of failures and runs, then checks each bucket against the bar the business set. No numpy, no scipy, nothing to install. Paste it into a file and run python3 signoff_bound.py.
"""How bad could it really be? Exact one-sided upper bound on a failure rate.
Standard library only. Usage: python3 signoff_bound.py"""
from math import comb
def binom_cdf(k, n, p):
return sum(comb(n, i) * p**i * (1 - p)**(n - i) for i in range(k + 1))
def upper_bound(failures, runs, confidence=0.95):
"""Highest failure rate still consistent with what we saw (Clopper-Pearson, one-sided)."""
lo, hi = 0.0, 1.0
for _ in range(60):
mid = (lo + hi) / 2
if binom_cdf(failures, runs, mid) > 1 - confidence:
lo = mid
else:
hi = mid
return hi
def runs_needed(target, failures_allowed=0, confidence=0.95):
n = failures_allowed + 1
while upper_bound(failures_allowed, n, confidence) > target:
n += 1
return n
# Fictional Northstar Systems refund agent, UAT results by bucket
buckets = [
# bucket, failures, runs, the bar the business owner set before UAT
("Routes ticket to the right queue", 1, 80, 0.10),
("Drafts refund reply a person sends", 2, 120, 0.10),
("Issues refund under $50 on its own", 0, 40, 0.01),
("Refuses refund on a closed order", 0, 150, 0.02),
]
print(f"{'bucket':38} {'seen':>9} {'could be':>9} {'bar':>5} verdict")
for name, f, n, bar in buckets:
ub = upper_bound(f, n)
verdict = "PASS" if ub <= bar else f"NOT YET (needs {runs_needed(bar, f)} runs at {f} fails)"
print(f"{name:38} {f:>3}/{n:<5} {ub:>8.1%} {bar:>5.0%} {verdict}")
And this is what it printed when I ran it:
bucket seen could be bar verdict
Routes ticket to the right queue 1/80 5.8% 10% PASS
Drafts refund reply a person sends 2/120 5.2% 10% PASS
Issues refund under $50 on its own 0/40 7.2% 1% NOT YET (needs 299 runs at 0 fails)
Refuses refund on a closed order 0/150 2.0% 2% PASS
Three things in that output are worth slowing down for.
The cleanest-looking bucket is the one that fails. "Issues refund under $50 on its own" had zero failures. It's the only bucket with a perfect record, and it's the only one that isn't ready. Forty clean runs can't support a 1% bar. That bucket needs about 300 clean runs, or a different design.
The bar is doing all the work. Routing passed with a failure in it, because a misrouted ticket is cheap and the bar is 10%. The automatic refund failed with no failures in it, because a wrong refund is money out the door and the bar is 1%. Same agent, same week of testing, opposite verdicts. The difference is entirely in what the business said it could live with.
The last row passes by a hair. 0 of 150 gives 1.98%, which rounds to 2.0% and sits just under the 2% bar. On a real project I'd write that down as "passes, no margin" and put it on the watch list for production, because one failure in the next 50 runs would push it over.
Turn the bar into a requirement
This is the BA part, and it's where I'd disagree with how most teams run this.
A lot of AI test plans describe the bar like this: "The agent should be highly accurate." Or, slightly better: "Target accuracy: 95%." Neither is testable. The first is a wish. The second sounds precise, but it doesn't say which cases, what counts as a failure, or how sure you need to be.
A testable version names four things for each bucket:
- The bucket. One kind of case with one kind of consequence. "Issues a refund under $50 without a person approving it."
- What counts as a failure. Written so two testers agree. "Refund issued when the policy says no, or refund amount differs from the policy amount by any amount."
- The bar. The highest failure rate the business owner will accept, written as a number. "1%."
- The confidence. Usually 95%. Say it out loud so nobody assumes 50%.
Then the sample size isn't a debate. It falls out of the math. A 1% bar at 95% confidence with zero failures allowed means about 300 runs. If the team can't afford 300 runs for that bucket, that's a design conversation, not a testing one: maybe refunds under $50 go to a person for the first month, and the agent's decisions get logged and compared until there's enough evidence.
Here's the acceptance criterion written that way, in the Given/When/Then shape I use in my acceptance criteria template:
Scenario: Agent issues small refunds within policy (bucket R1)
Given the R1 golden set of 300 refund requests under $50,
each with an expected decision set by the refunds lead before UAT
When the agent processes every request once, in a fresh session each time
Then no more than 0 decisions differ from the expected decision
And the exact 95% upper bound on the failure rate is at most 1%
And every run ID and decision is stored in the UAT evidence log
That last line matters more than it looks. If a reviewer can't open the runs, the count on the slide is a claim, not evidence.
A sign-off table you can copy
Here's the table I'd put on the sign-off page instead of "20 runs, 0 failures." Filled with the fictional Northstar numbers from the script:
| Bucket | What counts as a failure | Bar (set before UAT) | Runs | Failures | Point estimate | Could be as high as (95%) | Verdict | Owner |
|---|---|---|---|---|---|---|---|---|
| R1 Routing | Ticket lands in the wrong queue | 10% | 80 | 1 | 1.3% | 5.8% | Pass | Support ops lead |
| R2 Draft reply | Reply states the wrong policy or amount | 10% | 120 | 2 | 1.7% | 5.2% | Pass | Support ops lead |
| R3 Auto-refund under $50 | Refund issued against policy, or wrong amount | 1% | 40 | 0 | 0% | 7.2% | Not yet: needs 299 clean runs | Refunds lead |
| R4 Refuse closed order | Refund issued on an order past the window | 2% | 150 | 0 | 0% | 2.0% | Pass, no margin | Refunds lead |
Two columns do most of the work. "Bar (set before UAT)" stops the bar from drifting to fit the results, a temptation on any project, not just AI ones. And "Could be as high as" replaces the comforting point estimate with the number a risk owner actually needs.
If you'd rather keep the results in a database than a spreadsheet, the same check runs as SQL for the zero-failure case. The rule-of-three bound is just 3 divided by the run count:
-- One row per UAT run: bucket, run_id, passed (1/0)
SELECT bucket,
COUNT(*) AS runs,
SUM(1 - passed) AS failures,
ROUND(100.0 * SUM(1 - passed) / COUNT(*), 1) AS point_pct,
CASE WHEN SUM(1 - passed) = 0
THEN ROUND(300.0 / COUNT(*), 1) -- rule of three
ELSE NULL END AS zero_fail_bound_pct -- use the script when failures > 0
FROM uat_runs
GROUP BY bucket
ORDER BY bucket;
I left the failures-greater-than-zero case to the script on purpose. You can approximate the exact interval in SQL, but it's ugly, and an ugly query on a sign-off page invites someone to "simplify" it later.
The mistakes that make the number lie
The math is only honest if the runs are. These are the ways I'd expect a sharp tester or a skeptical risk owner to poke holes in a count, and they're all fair.
Runs that aren't independent
The formula assumes each run is a fresh draw. If you run the same 20 cases five times each and call it 100 runs, you haven't got 100 independent trials. You've got 20 cases, each measured five times. Those repeats tell you about stability (does case 14 flip between approve and reject?), which matters, but they don't tell you much about cases you haven't tried.
What I'd do: count distinct cases for the bound, and use the repeats separately as a stability check. A case that gives different answers across runs is a defect on its own, whatever the bound says. That's the "three runs" gate from my AI agent test plan, and it stays.
Easy cases padding the count
If 280 of your 300 refund cases are the boring kind (in policy, clear amount, one item), the bound tells you about boring cases. It says very little about the partial return with a promo code applied, which is where refund logic usually breaks.
What I'd do: split the bucket. If the hard cases have a different consequence or a different failure pattern, they're a different bucket with their own count. The bound is only as good as the definition of the bucket it's counting.
The test changed after the results came in
The fastest way to get a clean count is to decide, after the fact, that a failure "wasn't really a failure." Sometimes that's right: the expected answer in the golden set was wrong. But if it happens after the agent disagreed, write it down as a label change, with who changed it and why, and rerun that case. Don't quietly flip it.
Grading the transcript, not the world
Woliński's post makes a point I'd add to every AI UAT plan: grade the end state, not the agent's description of what it did. If the agent says "refund issued for $42.10," check the refunds table, not the message. An agent that describes the right action and takes the wrong one is a failure, and a transcript-only check will count it as a pass.
Confusing "rare" with "safe"
A 1% failure rate on refunds is rare per case. At 2,000 refunds a week it's about 20 wrong refunds a week. The bar should come from the weekly cost, not from how small the percentage sounds. When I ask a business owner for a bar, I ask it both ways: "what percentage?" and "how many wrong ones a week can you live with?" The second question usually gets a smaller number.
Roll a fair die 20 times without a six and you still can't conclude it has no six. AI test runs work the same way. Image: "Three six-sided dice.jpg" by PierreSelim, CC BY 3.0, via Wikimedia Commons.
What this method doesn't cover
I want to be plain about the limits, because a confidence bound can look more authoritative than it is.
- It assumes the test cases look like production. If real customers send messier requests than your golden set, the bound is about your golden set, not your customers. Keep sampling in production.
- It doesn't cover drift. A model update, a prompt change or a new tool can change the failure rate overnight. The bound is a fact about one version. Write the version next to the count.
- It treats every failure in a bucket as equal. A refund that's $0.50 off and one that's $500 off both count as one failure. If they matter differently, split the bucket.
- 95% isn't magic. It's a convention. For high-stakes, no-undo actions, some teams will want 99%. At 99% the rule of three becomes roughly 4.6 divided by n, per the same Wikipedia page, so the run counts go up by about half.
What I'd do on Monday
If I were handed an AI feature heading into UAT this week, this is the order I'd work in.
- List the buckets by consequence, not by feature. "Things it does alone that can't be undone," "things a person reviews," "things it should refuse." Each gets its own row.
- Get a bar from each owner, in writing, before the first run. Ask for a percentage and a weekly count. Put both in the UAT plan's entry criteria.
- Run the script on the bars, not the results.
runs_needed(0.01)tells you 299 before you've tested anything. If the team can't afford it, change the design now, while it's cheap. - Build the golden set to that size, with expected decisions set by the business owner, not generated from what the agent did.
- Replace "X runs, 0 failures" on the sign-off slide with the table above: bar, runs, failures, point estimate, the 95% bound, verdict, owner.
- Keep the evidence openable. Run IDs, inputs, outputs, and the end-state check for each.
- Rerun the golden set on every model or prompt change and update the table. Same buckets, same bars.
None of this needs a data scientist. It needs a BA who's willing to ask the business owner one uncomfortable question and write the answer down as a number.
If you're also deciding which AI tool to buy in the first place, the same idea applies earlier: set the pass criteria before the demo. And if the agent writes SQL, the count isn't enough on its own; review the SQL itself against a fixture with a known answer. For the plan around all of this, my notes on what a good UAT plan looks like cover entry and exit criteria.
Questions for you
- Have you ever seen an AI feature signed off on a count like "20 runs, 0 failures"? What was the bar, if anyone wrote one?
- Who on your team should own the failure bar for an AI feature: the product owner, the risk owner, or the person whose budget pays for the mistakes?
- Would you count repeated runs of the same case toward the sample size? I say no for the bound and yes for stability. Tell me where you'd draw it.
If you've run AI UAT with a real sample-size calculation, I'd like to hear how you got the business to commit to a number. Leave a comment.
References
- Shunyu Yao, Noah Shinn, Pedram Razavi, Karthik Narasimhan, τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, arXiv, submitted 17 June 2024.
- Wikipedia, Rule of three (statistics).
- Wikipedia, Binomial proportion confidence interval (Clopper-Pearson interval).
- Michał Woliński, How to Evaluate AI Agents: Grade the World, Not the Transcript, Dev.to, 12 November 2026, and the comments by aiapikey on the same page.
- Prachit Parikh, AI agent test plan template, BA Toolkit on GitHub.
About me
I'm Prachit Parikh, a business analyst. I turn what stakeholders ask for into requirements a team can test, check the data with SQL, and see changes through UAT. I keep free templates, including an AI agent test plan with pass marks you set before the first run, in my BA Toolkit on GitHub, and I make short animated explainers on YouTube. Besides Hashnode, I write on Substack, Medium, Dev.to, Blogger and WordPress, and I answer questions on Quora. You can also find me on LinkedIn, Bluesky, Kaggle, Credly, Crunchbase and Gravatar. Everything is collected on my website, my portfolio and my profile hub.
About the author
- Prachit Parikh is a business analyst. I turn what stakeholders ask for into requirements a team can test, check the data with SQL, and see changes through UAT. I write practical guides on requirements, UAT planning, data validation and Power BI reporting. Videos: https://www.youtube.com/@PrachitParikh1 More at https://sites.google.com/view/prachit-parikh
Comments
Join the discussionNo comments yet. Be the first to comment.
