This post is for engineering leaders who need to ship prompt and model changes without breaking what already works. It explains what an LLM eval harness consists of, which metrics to track, how to score without a human on every change, and how to keep the golden set honest.
Almost every production AI failure we have audited at customers (hallucinations leaking into customer chat, voice agents going off-script, document extractors emitting malformed JSON) traces back to the same missing piece: there was no eval harness, or there was one but nobody ran it before deploys. The model that worked in the prompt playground started misbehaving the moment real users hit it, and the team had no way to know which prompt change broke what.
An LLM eval harness is the contract between your AI system and the business outcome it is supposed to deliver. It is also the only thing that makes shipping prompt and model changes safe. Here is what it actually consists of, written from the deployments where we have built one.
1. The eval harness is a test suite, not a benchmark
Public benchmarks like MMLU, HellaSwag, and GSM8K measure general-purpose model capability. They do not measure your system. A model that scores well on MMLU can still mishandle a large share of your billing queries, because your billing queries depend on terminology, tone, and retrieval that the public benchmark does not contain.
An eval harness for your system is built from your data: 200 to 500 representative queries drawn from real user transcripts (or, before launch, your team's best simulation of them), each with the correct answer, the correct cited source, the correct tool to call, and a refusal flag if the query is out of scope. This is the golden set. Every prompt change, every model swap, every chunking tweak runs against the golden set before merging.
2. Pick the right metrics, and report them per category
A single quality score is almost always misleading because failures cluster in specific categories. The standard metric stack for a RAG chatbot looks like this, with each metric measured separately and reported by intent category:
- Retrieval recall@k: did the top k retrieved chunks include the source that contains the answer?
- Citation accuracy: does the cited source actually support the answer?
- Refusal correctness: did the system refuse out-of-scope queries and answer in-scope ones?
- Tool-call validity: did the agent call the right tool with the right arguments?
- Hallucination rate: the fraction of answers that contain a fact not supported by any retrieved chunk.
- Latency p50 and p95: end-to-end response time per turn.
- Cost per turn: token consumption multiplied by model price.
3. Auto-score where possible, human-score where it matters
Running every eval prompt past a human reviewer does not scale. But naive auto-scoring (for example, exact-string match against the golden answer) misses paraphrases that are equally correct. The pragmatic split: use deterministic auto-scoring for retrieval recall, citation correctness, tool-call validity, refusal correctness, latency, and cost, because these are objective. For answer quality on open-ended generations, use an LLM-as-judge with a clear rubric, validated against a sample of human-scored examples.
LLM-as-judge is cheaper and faster than human review, and once validated it tracks human scoring closely on most tasks. Validation means: take 50 golden examples, have a human score them, have the judge LLM score them, and compute Cohen's kappa. If kappa is above 0.7, ship the judge. If not, refine the rubric until it is.
4. Gate every release on the harness
The eval harness only adds value if it actually blocks deploys. Wire it into CI: any pull request that changes a prompt, retrieval config, model selection, or tool definition triggers a full eval run. The run reports the delta from the previous version on every metric. If any metric regresses past threshold (for example, citation accuracy drops more than 2 percentage points), the deploy is blocked.
This is the boring, unglamorous discipline that separates AI systems that ship reliably from AI systems that ship once and then quietly decay over months of "just one more prompt tweak." With CI-gated evals, every change has measured impact. Without them, every change is a roll of the dice.
5. Run continuous evals on production traffic, not just CI
CI evals catch known regressions. Continuous production evals catch unknown ones: the shifts in user behavior, corpus drift, or model-provider changes that you did not anticipate. Sample 5 to 10 percent of production traffic, run it through the eval rubric (LLM-as-judge), and alert when category-level metrics shift more than two standard deviations from baseline.
- Daily evals on full production traffic for low-volume systems.
- Sampled hourly evals on high-volume systems (chatbots handling 10k or more conversations a day).
- Per-customer eval slices for multi-tenant SaaS, to catch a single tenant's corpus shift breaking their experience.
- Drift alerts that fire on metric shifts, not just absolute thresholds, so you catch slow degradation.
6. Keep the golden set honest
The single biggest mistake teams make with eval harnesses is letting the golden set rot. New product features ship, the user behavior distribution changes, and the eval set keeps testing what last quarter's product did. Six months in, the harness reports 99% on a set of queries nobody actually asks anymore.
Refresh the golden set on a cadence: quarterly is a reasonable default for most systems, monthly for fast-moving products. Sample new real-user queries (with PII redacted), have a human review them, add the high-signal ones to the golden set, and retire the obsolete ones. The harness should evolve with the product, not lock it into a past version of itself.
Wrapping up
An eval harness is the load-bearing piece of production AI infrastructure. It is what lets you ship prompt changes confidently, swap models without dread, and prove to a regulator that your system did what you said it would. Build it in week one of any LLM development engagement, gate every release on it, and refresh it quarterly. The teams that do this ship AI that survives past the demo. The teams that skip it ship a brittle prototype and rebuild it six months later.
If you have a system in production without an eval harness, or one that nobody runs before deploys, get in touch. We will review what you have and propose a golden set, metric stack, and CI gate sized to the workload.
Frequently asked questions
How large should a golden eval set be?
200 to 500 representative queries drawn from real user transcripts (or a careful simulation before launch), each with the correct answer, cited source, tool call, and a refusal flag where the query is out of scope.
When is LLM-as-judge reliable enough to use?
After validation: score 50 golden examples with both a human and the judge model and compute Cohen's kappa. Above 0.7, ship the judge; below that, refine the rubric first. Use deterministic scoring for objective metrics like recall, citation, tool-call validity, latency, and cost.
How often should the golden set be refreshed?
Quarterly for most systems, monthly for fast-moving products. Sample new real-user queries with PII redacted, have a human review them, add the high-signal ones, and retire obsolete ones.