How to test AI output before you put it in front of a customer

Build a small set of questions with known-correct answers, run it on every change, and measure the failures you actually care about.

Teams test AI by trying it. Someone asks a few questions, the answers look good, and it ships. Then the model version changes, or a prompt gets tidied, and nobody notices for six weeks.

You need a regression suite. It is smaller and duller than people expect.

Start with twenty questions, not two hundred

Twenty real questions, with answers you already know to be correct, beats a large synthetic set. Sources worth mining:

the questions the team actually asked in the first two weeks
every case where it got something wrong (these are the most valuable)
the three questions your most sceptical colleague uses to dismiss it

Write the expected answer next to each. Where an exact answer is not meaningful, write the property that must hold: "cites at least one document", "does not give a figure", "says it does not know".

Grade the things that matter, not the wording

Most output is prose, and prose comparison is a trap. Grade:

Check	How
Are the figures right?	Exact match — these came from code, so they must be exact
Did it cite a source?	Presence of a document id or title
Did it use the right records?	Compare the set, not the sentence
Did it refuse when it should?	Include questions with no answer in the data
Is the recommendation the same?	Categorical, not phrasing

Include questions that should fail. A suite made only of answerable questions measures nothing about the failure you care about most. Roughly one in five of ours has no correct answer available — the pass condition is that the system says so.

Run it on every change, and know which change

Prompt edits, tool changes, retrieval changes, and model version upgrades all break things. The last is the most common cause of "it used to work" and the hardest to notice, because nothing in your system changed.

Record the score with the date, the model version and the prompt version. A number without those three is not comparable to anything.

Grading with a model, honestly

For anything not exactly checkable, a second model can grade — with two caveats people skip:

Give it the reference answer, not just the question. Asking a model whether an answer is good, without telling it what good is, measures fluency.
Calibrate it once by hand. Grade thirty items yourself, compare, and find out how often the grader agrees with you. If it agrees 70% of the time, your suite has a ±30% error bar and you should stop quoting it to two decimals.

The number that actually matters

Not accuracy. How often it is confidently wrong — a fluent, well-formatted, authoritative answer that is false. Everything else is recoverable; that one reaches a customer.

Track it separately, and set the bar where the business actually needs it. For an internal drafting aid, some tolerance is fine. For anything a customer sees unedited, near zero is the only acceptable number, and if you cannot demonstrate it, keep a human in the loop and stop calling that a temporary measure.

When to stop testing and ship

When the suite passes, and someone who did not build it has used it for a week on real work without finding a new failure class. That week finds things no suite does — and each thing it finds becomes question twenty-one.