How to Test AI Output Quality Without Eyeballing It

Testing an AI feature means running a frozen set of real inputs through it, scoring each one with code, and comparing versions case by case rather than by average. Eyeballing a few outputs cannot catch the failure that matters: an update that raises your overall score while quietly breaking cases that used to work. Build the golden set first, score cheaply before you score cleverly, and write your shipping threshold before you look at the result.

Eyeballing fails in one specific, documented way#

The problem with spot-checking is not that you look at too few outputs. It is that the aggregate number you fall back on is actively misleading.

A 2023 study of five GPT-3.5-family models found that in 87.9% of API updates where overall accuracy improved, at least one previously correct prediction still regressed. Individual predictions regressed in 10.9% of cases across those updates. A team watching the average would have shipped every one of them.

The same study shows the effect is not even consistent within one update. Moving from gpt-3.5-turbo-0301 to gpt-3.5-turbo-0613 on the Civil Comments dataset dropped one prompt's accuracy by 9.6% while lifting another prompt's by 5.1% — same model change, same data, opposite verdicts depending on which prompt you happened to test.

What a golden set is made of, and where cases come from#

Pull cases from production logs, not from imagination. Inputs you invent cluster around what you expect users to do, which is precisely the distribution your feature already handles.

Fill three buckets. First, a representative sample of ordinary traffic — the boring middle, because a suite of only hard cases cannot tell you when the easy ones break. Second, every ambiguous input where two people on your team disagreed about the right answer. Third, one case for every bug you have already shipped.

Anthropic's guidance on building evaluations says to be task-specific and mirror the real-world task distribution, and names the edge cases worth including: irrelevant data, overly long input, harmful input, ambiguous cases. Its more counterintuitive principle is to prioritize volume over quality — in its own words, more questions with slightly lower signal automated grading beats fewer hand-graded ones.

Freeze the set and version it in git next to the prompt. If cases drift while the prompt changes, every comparison you run afterward is meaningless.

Four ways to score, ranked by how much you should trust them#

Score cheaply before you score cleverly. Most teams reach for a model-based judge first and never build the deterministic checks that would have caught the structural failures for free.

OpenAI's grader documentation gives a useful neutral vocabulary for tiers 1 and 2: string check graders returning binary 0 or 1, text similarity graders, score model graders, custom Python graders, and a multigrader that combines several by formula. Anthropic's docs map the same territory from the other side with exact match, cosine similarity, ROUGE-L, and model-based grading.

The highest-value check in tier 2 is the grounding assertion: take every quoted span in the output and confirm it appears literally in the input. It is a substring search, it costs nothing, and it catches invented citations that pass every schema check you own. For the prompt-side half of this — writing output contracts a validator can actually enforce — see prompts that survive model updates.

The human rubric you cannot write in advance#

Do not book a meeting to write the rubric. Shankar and colleagues named the trap precisely: you need criteria to grade outputs, but grading outputs is how you discover the criteria. A rubric written before anyone has read real output will be wrong.

Their finding goes further. Some criteria turn out to depend on the specific outputs observed rather than being universally applicable, so the rubric is partly a discovery artifact of your own system's behavior.

The working procedure is short. Draft a rough rubric, grade 20 real outputs against it, then rewrite the rubric with what you learned — and expect to delete at least one criterion that sounded good in the abstract.

1. Two independent raters, no discussion

   Both grade the same 30 to 50 cases without seeing each other's scores. Disagreement rate is information; consensus reached by talking first is not.

2. Binary per criterion, never a 1-to-5 vibe

   "Does the summary contain a fact absent from the source?" has an answer. "Rate the quality 1-5" does not survive two raters.

3. Adjudicate every disagreement into a written rule

   Each resolved disagreement becomes one sentence in the rubric. That sentence is the actual deliverable.

4. Keep the labeled set as ground truth

   This is not overhead. It is the yardstick you will measure your automated judge against in the next section.

What an LLM judge actually gets wrong#

A judge model is an instrument with known, measured defects. Read these before you put one in a merge gate.

Position bias. Wang and colleagues showed that merely changing the order in which two responses were shown to the judge could make Vicuna-13B beat ChatGPT on 66 of 80 tested queries. Same responses, different order, opposite verdict.

Rubric-order bias, which is worse. A 2026 paper found that rubric scoring behaves like multiple-choice selection, so the order of the rubric options itself introduces systematic bias. All six models tested showed position bias — significant across the board on the two human-rated benchmarks — and the top-ranked candidate flipped on 16% to 39% of prompts between balanced and fixed orderings.

Self-preference. Panickssery and colleagues found LLM evaluators score their own outputs higher than other models' outputs on text human annotators rated as equal quality, with a linear relationship between self-recognition ability and self-preference strength. Never let the model that generated the output grade it.

Verbosity and leniency. Zheng and colleagues named position, verbosity, and self-enhancement bias plus limited reasoning ability as the judge failure modes in MT-Bench. Thakur and colleagues tested thirteen judge models and found only the largest reached reasonable human alignment, still well behind inter-human agreement, with a general tendency toward leniency.

Percent agreement is a liar. The same thirteen-judge study found judges with high percent agreement can still assign vastly different scores, with the best judges up to 5 points off human scores on the scales used. And OpenAI's own grader docs name grader hacking: a model can learn to score well on your grader while scoring poorly under human expert review.

Calibrate the judge before you believe a single number it prints#

Turn the two previous sections into a procedure you run once per judge version.

Hold out the human-labeled cases. Run the judge over them blind and report judge-versus-human agreement as a number you publish alongside every eval result that used the judge. If you cannot state that number, your eval has no error bar at all.

A minimal eval harness, no dependencies#

Here is the whole thing in Python standard library only. The model call sits behind one function so it works with any provider, and the output that matters is the per-case file, not the summary number.

# eval.py — dependency-free eval harness. Python 3.9+.
# Usage:  python eval.py run cases.jsonl prompt.txt out_new.jsonl
#         python eval.py compare out_old.jsonl out_new.jsonl
import json, re, sys, time
from pathlib import Path

FLIP_BUDGET = 0   # max pass->fail regressions you will merge

# ---- the only provider-specific code in this file ----
def call_model(prompt, case_input):
    """Return (text, input_tokens, output_tokens).
    Replace the body with your SDK call and read token counts off the response."""
    raise NotImplementedError

# ---- graders: each returns (bool, reason) ----
def grader_schema(out, case):
    try:
        data = json.loads(out)
    except json.JSONDecodeError as err:
        return False, "not JSON: %s" % err
    missing = [k for k in case["required_keys"] if k not in data]
    return (not missing), ("missing: " + ",".join(missing) if missing else "ok")

def grader_grounded(out, case):
    """Every quoted span must appear verbatim in the input."""
    quotes = re.findall(r'"([^"]{12,})"', out)
    bad = [q for q in quotes if q not in case["input"]]
    return (not bad), ("unsupported: " + bad[0][:40] if bad else "%d grounded" % len(quotes))

def grader_enum(out, case):
    ok = out.strip() in case["allowed"]
    return ok, "in allowed set" if ok else "got: " + out.strip()[:40]

GRADERS = {"schema": grader_schema, "grounded": grader_grounded, "enum": grader_enum}

def load(path):
    lines = Path(path).read_text(encoding="utf-8").splitlines()
    return [json.loads(x) for x in lines if x.strip()]

def run(cases_path, prompt_path, out_path):
    prompt = Path(prompt_path).read_text(encoding="utf-8")
    rows = []
    for case in load(cases_path):
        start = time.perf_counter()
        text, tok_in, tok_out = call_model(prompt, case["input"])
        ms = round((time.perf_counter() - start) * 1000)
        scores = {}
        for name in case.get("graders", ["schema", "grounded"]):
            ok, why = GRADERS[name](text, case)
            scores[name] = {"pass": ok, "why": why}
        rows.append({
            "id": case["id"],
            "cluster": case.get("cluster", case["id"]),
            "output": text, "scores": scores,
            "input_tokens": tok_in, "output_tokens": tok_out, "ms": ms,
            "pass": all(s["pass"] for s in scores.values()),
        })
    Path(out_path).write_text("\n".join(json.dumps(r) for r in rows), encoding="utf-8")
    print("wrote %d cases to %s" % (len(rows), out_path))

def compare(old_path, new_path):
    old = {r["id"]: r for r in load(old_path)}
    new = {r["id"]: r for r in load(new_path)}
    ids = sorted(set(old) & set(new))
    n = len(ids)
    broke = [i for i in ids if old[i]["pass"] and not new[i]["pass"]]
    fixed = [i for i in ids if not old[i]["pass"] and new[i]["pass"]]
    rate = lambda d: sum(d[i]["pass"] for i in ids) / n
    p95 = lambda d: sorted(d[i]["ms"] for i in ids)[max(0, int(0.95 * n) - 1)]
    out_tok = lambda d: sum(d[i]["output_tokens"] for i in ids)
    print("n=%d  old=%.1f%%  new=%.1f%%" % (n, 100 * rate(old), 100 * rate(new)))
    print("regressions pass->fail: %d %s" % (len(broke), broke))
    print("fixes fail->pass:       %d %s" % (len(fixed), fixed))
    print("p95 ms:      old=%d new=%d" % (p95(old), p95(new)))
    print("out tokens:  old=%d new=%d" % (out_tok(old), out_tok(new)))
    sys.exit(1 if len(broke) > FLIP_BUDGET else 0)

if __name__ == "__main__":
    cmd = sys.argv[1]
    run(*sys.argv[2:5]) if cmd == "run" else compare(*sys.argv[2:4])

Each line of cases.jsonl carries an id, the frozen input, the graders to apply, and whatever ground truth those graders need. The cluster field matters more than it looks — it is how you mark cases drawn from the same source document, which the sizing section explains.

How big does the set actually have to be#

Most golden-set advice stops at "build a set, then compare the scores." It does not tell you the error bar on the number you just compared.

For binary pass/fail scores on independent questions, the standard error of the mean is SE = sqrt(s * (1 - s) / n), where s is your observed pass rate. The general sample-size formula is n = (z_alpha/2 + z_beta)^2 * variance / delta^2.

Evan Miller's 2024 Anthropic paper works a full example. Under stated assumptions — binary scores, uniform question difficulty, variance of conditional means of 1/9, zero conditional variance, 80% power, 5% significance — detecting an absolute difference of 3 percentage points needs roughly 969 independent questions. The paper suggests new evals contain at least 1,000.

Independence is the other trap. When several cases come from one shared document or ticket thread, they are correlated, and Miller's paper recommends clustered standard errors — its measured examples came out 3.05x larger than naive ones on DROP and 1.88x larger on MGSM. Ten test cases from one long customer email are not ten independent observations.

Compare per case, not per average#

This is the payoff, and it is the whole reason the harness writes a per-case file. Miller's fourth recommendation is to conduct statistical inference on question-level paired differences when two models are being compared.

The practical version is a flip table. Run old and new over the same frozen set, then count regressions and fixes separately. Six regressions and eight fixes nets out to plus two, which is exactly what a clean two-case improvement looks like on an average — and they are completely different decisions.

That gap is the 87.9% finding in operational form. The regressions are real, they are named, and now they are on the pull request instead of in your inbox next Tuesday.

Handle nondeterminism here too. A single sample per case above temperature zero is a coin flip; Miller's third recommendation is to reduce variance by resampling answers. In his uniform-difficulty example, two samples per question cut variance by a third and six cut it by five-ninths, with diminishing returns after that.

Wire it into CI so a prompt diff cannot merge silently#

Every pull request that touches a prompt, a model version, or a retrieval config runs the suite. The flip table gets posted as a comment on the pull request. A regression count over your flip budget exits non-zero and blocks the merge.

A small suite is enough to catch large damage. A 2026 paper by Daniel Commey using 30-case suites on Llama 3 8B Instruct and Qwen 2.5 7B Instruct recorded Qwen falling from 26/30 to 9/30 on a retrieval citation-compliance task purely from appending generic quality rules to the user prompt — a 56.7-point drop. Tightening the output contract, rather than adding rules about quality, lifted strict extraction from 0/30 to between 28/30 and 30/30 on another task.

Keep the pull request run under a few minutes or people will route around it. Push the large nightly sample to asynchronous batch processing — Anthropic's pricing page documents a 50% discount on both input and output tokens for its Batch API, and cache reads billed at 0.1x the base input rate (checked August 2026, check the page for current terms). Reuse one cached system prompt across every case and the nightly run stops being a budget conversation. The trade is turnaround: Anthropic's docs say most batches finish inside an hour, but a batch may take up to 24 hours and expires if it has not completed by then — which is exactly why this belongs in the nightly job and never in the pull request gate.

Measure cost and latency in the same table as quality#

Quality alone is not a shipping decision. A prompt change that adds four points of accuracy and doubles output length is a trade, and you cannot see the trade unless both numbers land in the same row.

Log tokens per case from the API response. Anthropic's Messages API returns input_tokens, output_tokens, cache_creation_input_tokens, and cache_read_input_tokens, and its docs state that total input tokens is the sum of all three — so a harness recording only input_tokens under-reports cost on every cached prompt. For turning those counts into a monthly figure, see LLM API cost estimation.

For latency, Anthropic's docs define baseline latency as time to process the prompt and generate the response, and time to first token as the gap between sending and the first token arriving — the one that matters when you stream. Report p95, not mean; the mean is where your worst cases go to hide.

Deciding to ship#

Write the threshold before you look at the result. That single habit is what stops a team from rationalizing a bad run at 6pm on a Thursday.

Use three gates rather than one composite score:

1. Zero new hard failures. No case that used to produce structurally valid output now fails to. This gate is absolute and has no budget.
2. A structural validity floor. State the minimum tier-2 pass rate for the suite as a whole and do not negotiate it downward in the pull request.
3. A flip budget. A stated maximum number of pass-to-fail regressions you will accept, with each one reviewed and signed off by a named human.

Two tiebreakers settle the arguments. When the judge and the human labels disagree, the humans win and the judge goes back for recalibration. When the flip count sits inside your noise, you have not measured an improvement — you have measured nothing, and shipping is a coin flip you are calling heads.

Ship behind a flag and keep sampling production into the golden set. The frozen set stops representing reality the moment your traffic shifts, and it will. Pair this with real failure handling — retries, idempotency, and alerting — so a bad case in production surfaces as a signal rather than a silence.

Who this is not for#

If a human reads every output before it reaches a customer, re-reading is your quality process and this whole apparatus is ceremony. Add tier-1 and tier-2 graders, pin your model version, and stop.

Below a low volume of automated calls, structural validation plus a pinned version is a fair place to stop too. Skip the judge, skip the statistics, and spend the time on the prompt instead.

The full machinery earns its keep when output feeds another system with no human in between — routing, extraction, classification, ranking — and when the cost of a silent wrong answer is paid by someone who is not you. If you are still choosing the underlying approach, RAG versus fine-tuning is the decision that comes before this one.

Build the flip table first#

If you do one thing from this article, make it this: write the per-case results file and the compare command before you write a single grader beyond json.loads.

Averages are the reason teams ship regressions with a smile. The flip table is one short function and it converts "the score went up" into "these six cases broke, here are their IDs, here is who signed off." That is a decision you can defend.

Then add graders in trust order — exact match, structural, grounding, human labels, judge last and only after you have measured what it agrees with. Every tier you add before the flip table is a more precise measurement of a number you are still reading wrong.