"It looked great in the demo" is not an eval strategy.
Every production AI system we've worked with had a version of the same story: responses looked good during development, the team deployed with confidence, and then real users found edge cases that produced wrong answers, hallucinated facts, or inappropriate responses within the first week.
The problem isn't that the LLM is bad. The problem is that "looked great" is a binary eyeball test on a tiny sample. Production systems need a measurement framework that catches quality regressions before users do.
The five evaluation dimensions
A complete LLM eval covers five distinct dimensions. Most teams evaluate one or two. All five are necessary for a production system.
1. Correctness
Is the answer factually accurate? This is the most intuitive dimension, but also the hardest to automate at scale.
For closed-domain questions (your knowledge base contains the answer), correctness is testable: did the system return the right answer? You can build a labeled test set and compute exact-match or F1 scores.
For open-domain questions or tasks that require synthesis, correctness requires LLM-as-judge evaluation (more on this below) or human review.
Correctness failures are the most damaging to user trust. A system that gives wrong answers confidently loses users fast.
2. Groundedness
Did the response stay grounded in the source material? This applies specifically to RAG systems. A grounded response is one where every factual claim can be traced back to a retrieved chunk.
Groundedness and correctness are not the same. A response can be factually correct but ungrounded (the LLM drew on its training data rather than the retrieved context). In regulated industries — healthcare, legal, financial — this distinction matters enormously.
Test for groundedness by verifying that key claims in the response appear verbatim or near-verbatim in the retrieved chunks.
3. Relevance
Did the system answer the question that was actually asked? Irrelevant responses — responses that technically say true things but don't address the user's query — are a common failure mode in RAG systems that retrieve broad chunks and let the LLM ramble.
Relevance evals compare the question to the response and score how directly the response addresses the query. This is a natural fit for LLM-as-judge.
4. Safety
Did the response violate any safety or policy constraints? For customer-facing systems, this covers: off-topic responses, competitive mentions, regulatory claims, PII exposure, and anything that violates your brand or legal guidelines.
Safety evals are not optional for production systems. Even if your system passes all other dimensions, a single inappropriate response surfaced in a screenshot causes disproportionate damage.
Run safety evals as a separate dimension with a hard pass/fail gate, not as a soft score.
5. Format compliance
Did the response match the expected format? This sounds trivial, but it's a source of real failures in automated pipelines. If downstream systems expect a JSON object and the LLM returns a JSON object wrapped in a markdown code block, the pipeline breaks.
Format evals are fully automatable — parse the output, check the schema, pass/fail.
Building a test case dataset
A useful eval harness starts with 100–300 test cases. Each case has:
- Input: the user message (or the full prompt, including context)
- Expected output: what a correct, high-quality response looks like
- Eval type: which dimensions apply to this case
- Category: the type of query (so you can slice results by category)
Where do these cases come from?
- Manually curated examples — write cases that cover your core use cases plus known edge cases
- Production logs — once you have any user traffic, sample real queries (with PII stripped)
- Red-teaming — generate adversarial cases that probe failure modes: leading questions, ambiguous inputs, requests outside scope
- Regression cases — every bug that makes it to production becomes a test case
Don't aim for perfection at 50 cases. A rough set of 100 cases run consistently is more valuable than a perfect set of 1,000 cases you run once.
Automated eval with LLM-as-judge
For dimensions that require judgment (correctness on synthesis tasks, relevance, groundedness), you can automate scoring by sending the input, response, and a rubric to a separate LLM and asking it to score.
A basic LLM-as-judge prompt:
You are evaluating a RAG system response.
Question: {question}
Retrieved context: {context}
System response: {response}
Score the response on the following criteria:
1. Groundedness (1-5): Are all factual claims supported by the context?
2. Relevance (1-5): Does the response directly address the question?
3. Correctness (1-5): Is the response factually accurate based on the context?
Return a JSON object with scores and a one-sentence justification for each.
LLM-as-judge is not perfect — it has its own biases and can be fooled by confident-sounding wrong answers. Use it for directional scoring and bulk regression testing. Use humans for calibration and edge cases.
Meta-eval your judge: take a sample of its judgments and have a human score them. If the judge agrees with humans 80%+ of the time, it's useful for automated regression. Below 70%, the judge needs a better rubric or a stronger model.
Human eval for edge cases
Some things LLM judges get wrong systematically:
- Responses that sound confident and fluent but are subtly incorrect
- Culturally sensitive content
- Domain-specific technical accuracy (medical, legal, financial)
- Responses that are technically correct but would confuse a real user
Reserve human eval for: high-stakes use cases, periodic calibration of your automated evals, and any case the automated eval flags as borderline.
Human eval doesn't need to cover every case — it needs to cover the right cases. A rotating sample of 20–30 cases per week, reviewed by someone who knows the domain, catches more systematic issues than trying to review everything.
Setting pass/fail thresholds
Scores are only useful if you act on them. Set explicit thresholds before launch:
| Dimension | Minimum to ship | Amber flag | Red flag | |---|---|---|---| | Correctness | 85% | 75–85% | Below 75% | | Groundedness | 90% | 80–90% | Below 80% | | Relevance | 85% | 75–85% | Below 75% | | Safety | 99% | 97–99% | Below 97% | | Format compliance | 99% | 95–99% | Below 95% |
These are starting points — your thresholds should reflect the risk profile of your use case. A healthcare RAG system should have tighter thresholds than an internal FAQ bot.
Run the full eval suite before every significant prompt change, every model version update, and on a weekly schedule against your test set. Scores that drift downward over time — even without any code changes — indicate data staleness or model drift.
The eval infrastructure you actually need
You don't need a sophisticated platform to start. A spreadsheet with test cases and a Python script that runs each case against your system and scores it is sufficient for the first 90 days. The important thing is having the discipline to run it and act on the results.
When your test set grows beyond 500 cases or you're running evals on multiple system variants simultaneously, look at purpose-built eval frameworks — Langfuse, Braintrust, or PromptLayer are the tools most teams end up using for AI agent development at scale.
Build the eval harness before you build the system. It will change how you design prompts, what test cases you catch, and how confidently you push to production.