You Don't Have a Ground Truth Set. Evaluate Anyway.

By Adrian Vance · Mining decisions humans already made, instead of authoring an answer key.

Every team that ships an LLM feature hits the same wall about six weeks in. The demo worked. The pilot users liked it. Then someone changes a prompt, and nobody can say whether the change made the product better or worse.

The instinct at that point is to build an eval set: a few hundred inputs, each paired with the correct output, scored automatically on every deploy. It is the right instinct. It is also where most teams stall, because writing the correct output for two hundred open-ended prompts is genuinely hard work that nobody has time for, and the person who would be best at it is the one whose calendar is most expensive.

So the eval set never gets built, and the team ships on vibes for another year.

The way out is not to write the answer key faster. It is to stop writing it at all.

Ground truth is a discovery problem, not an authoring problem

The framing error is treating the answer key as something you produce. In almost every product with an LLM in it, decisions of the exact type your model is making have already been made by humans, and somebody wrote them down.

A support-triage classifier does not need you to invent the right category for four hundred tickets. Your support team has been assigning categories to tickets for three years. That is a labeled dataset with human judgment baked in, sitting in your helpdesk export.

A summarization feature for meeting notes does not need hand-authored gold summaries. If your team keeps written recaps, the recaps are the summaries a human decided were worth keeping.

A code-review assistant does not need a rubric invented from scratch. Your merged pull requests contain thousands of instances of a reviewer deciding a change was acceptable, and your reverted commits contain instances of the opposite.

The pattern generalizes: find a place where a human already made the decision your model is being asked to make, and where the decision was recorded as a byproduct of normal work. That artifact is your ground truth. It is more honest than anything you would author in a conference room, because it was produced under real conditions with real consequences, not by someone imagining what a good answer looks like.

This reframe usually turns a three-week authoring project into a two-day extraction job.

Three failure modes when you mine existing decisions

Free ground truth is not clean ground truth. Three things go wrong reliably.

Selection bias in what got recorded. The tickets that got manually categorized are not a random sample of tickets. They are the ones that made it past the first-line filter. If you evaluate only on those, you will be blind to the easy cases your model might quietly break, and you will overestimate difficulty across the board. Check the distribution of your mined set against the distribution of live traffic before you trust a score.

Label drift over time. Category definitions move. A "billing" ticket in 2023 may be a "subscription" ticket in 2026 because someone reorganized the taxonomy. If you pull three years of history into one eval set, you have quietly built a set with two incompatible answer keys in it. Bound the window, or version the labels.

Disagreement that the artifact hides. A merged pull request records that one reviewer approved. It does not record that two others would have rejected it. Any single recorded decision is one draw from a distribution of plausible human judgments. Where you can, find cases with multiple independent decisions and measure how often humans agreed with each other. That number is your ceiling. A model scoring 85 percent against a key where humans only agree with each other 80 percent of the time is not underperforming, it is saturated, and further prompt tuning is measuring noise.

That last point is the one teams skip, and it costs them weeks. Without an inter-rater baseline you cannot distinguish "the model is wrong" from "the question does not have one right answer." Both look identical in a score.

Scoring: match the metric to the decision, not to the output

Once you have inputs and recorded decisions, you need a scorer. The temptation is to reach for semantic similarity between the model's text and the human's text. Resist it for anything where the output feeds a decision.

If the downstream use is a routing choice, score the choice. Exact match on the category, and report a confusion matrix rather than a single accuracy number, because the cost of misrouting a security escalation into billing is not the cost of misrouting billing into subscriptions. An aggregate accuracy figure averages over cost asymmetries that the business cares about a great deal.

If the output is prose that a human will read and act on, score the extractable claims rather than the phrasing. Pull the entities, numbers, and commitments out of both the model output and the human artifact, and compare those. Two summaries can share almost no wording and be equally correct; two summaries can be near-identical in embedding space and differ on the one number that mattered.

If you genuinely need a judgment call scored, an LLM judge is defensible, but only after you have validated the judge itself against the human-decided set. Run the judge on cases where you already know the recorded human decision and measure its agreement rate. If your judge agrees with humans 70 percent of the time, every downstream number it produces carries that error, and you should say so out loud in the dashboard rather than reporting a clean-looking percentage.

What this buys you

The point of the harness is not the score. It is that a prompt becomes a versioned artifact you can change deliberately.

Right now, on most teams, the system prompt is a paragraph somebody pasted in eight months ago that nobody will touch because nobody can prove a change is safe. That paragraph is production code with no test coverage. It gets edited in a hurry during an incident, and the edit is never evaluated.

With even a rough hundred-case set mined from recorded decisions, the loop closes. Change the instructions, rerun the set, see the delta, keep or revert. The absolute score barely matters. The delta is the whole product. A set that is 80 percent representative and available today will improve your system faster than a perfect set that arrives next quarter, because it lets you iterate now.

Start with fifty cases. Pull them from wherever your team already recorded a decision. Score the decision, not the prose. Measure how often your humans agree with each other before you judge your model against them.

Then change one thing in the prompt and watch the number move. That is the first time most teams find out whether their careful prompt engineering was doing anything at all.

---

Adrian Vance writes about applied AI, evaluation, and decision systems for technical audiences.

---

« back to writing