Can 100 examples approve your AI?
A team has reviewed a hundred AI outputs. The error rate sounds fine. Is that enough to ship?
An example is not a release decision.
Find the failure
A reviewer sees that an AI-written claim is unsupported by the source record. That is an error label.
This requires a clear rubric, source evidence, and people or systems qualified to apply it.
Decide whether to release
A product team aggregates reviewed labels, checks sample coverage and uncertainty, then compares the result with a policy.
This requires an explicit decision rule. It cannot be inferred from a model’s confidence score.
Try a release rule.
These are synthetic reviewed outputs from one intended workflow. The calculation is a two-sided 95% Wilson upper confidence bound: a way to avoid treating a small clean sample as proof of a low underlying error rate.
This lab’s illustrative policy requires at least 100 reviewed outputs. A real policy must also define the workflow, error class, sampling plan, reviewer agreement, and all other blocking rules.
- Observed error rate
- —
- Upper uncertainty bound
- —
- Policy limit
- —
What this does and does not mean
The formula asks: given the reviewed batch, how high could the true error rate plausibly be? It does not prove the model works in a different specialty, prompt format, or workflow. It cannot repair biased samples, missed labels, or a vague error definition.
The 100-output floor and policy limits here are teaching settings, not clinical safety standards and not a reproduction of an auditor’s claim-sampling method.
What should a company leader ask for?
Before the test
Name the intended workflow and unacceptable failure modes. Decide who labels cases and what makes a case representative.
At the gate
Require the reviewer evidence, the sample definition, the error count, the uncertainty bound, and the policy version together.
After release
Preserve failures as regression cases. Monitor the live workflow and decide in advance who can pause or roll back the system.