How do you evaluate an LLM application?
Short answer
Evaluate an LLM application with a versioned set of real test inputs, graded by code checks wherever possible and by rubric-based LLM judges, calibrated against human ratings, where not. Run it on every change to prompts, models, retrieval or tools, then keep watching production, because offline scores only predict live quality when the test set looks like real traffic. Measure the whole system, not the model alone.
By Dr. Dulan Dias, co-founder and CTO of Tellme AI
Key facts
- Zheng et al. (NeurIPS 2023) found that strong LLM judges such as GPT-4 agreed with human preferences more than 80% of the time, about the level at which humans agree with each other, and documented position, verbosity and self-enhancement biases.
- Anthropic's evaluation guidance recommends test sets that mirror the real task distribution, edge cases included, automated grading where possible, and a larger automatically graded set over a small hand-graded one.
- Shankar et al. (2024) named criteria drift: people need criteria to grade LLM outputs, but grading outputs is how they discover their criteria.
- RAGAS (Es et al., 2023) evaluates retrieval-augmented generation without ground-truth human annotations, scoring whether retrieved context is relevant and focused and whether the answer uses it faithfully.
- τ-bench (Yao et al., 2024) introduced pass^k, which measures how reliably an agent succeeds across repeated attempts at the same task, and scored agents on the final state of a database rather than on their transcripts.
- As of November 2026, OpenAI's hosted Evals platform is read-only and scheduled to shut down on 30 November 2026.
- From Model to Product recommends monitoring how often a language model is called, how long responses take, how often they are empty or invalid and how many tokens are used, because an API can return a valid response that is not useful.
Definitions
- Golden set
- A versioned collection of representative inputs, each paired with the expected answer or the properties a good answer must have, used to score every change to an LLM application.
- LLM-as-a-judge
- Using a language model to grade another model's output against a rubric, or to pick the better of two outputs. It scales human judgement but has biases of its own, so it must be calibrated against human ratings.
- Faithfulness
- Whether every claim in an answer is supported by the context the model was given. In retrieval-augmented systems it separates generation errors from retrieval errors.
- pass^k
- A reliability measure introduced with τ-bench: the chance that an agent succeeds on all k repeated attempts at the same task, rather than on at least one of them.
Start by defining what good looks like
An evaluation is only as useful as its definition of success, so write that down for each feature before choosing a metric. A support assistant might need answers that are correct under the policy, cite it, stay under a length limit and hand off when unsure; each is a separate, checkable criterion.
Expect the definition to change once you read outputs. Shankar et al. call this criteria drift: you need criteria to grade outputs, but grading outputs is how you discover your criteria. So begin with error analysis: read a hundred real outputs, note every way they fail, and turn the recurring failures into criteria.
Measure in layers. Chapter 1 of From Model to Product separates model metrics, system metrics such as latency and cost, and product metrics such as task completion, overrides and satisfaction, and warns that a model can improve while the product gets worse.
Build a golden set from real inputs
The golden set is the core asset: real inputs, each paired with the expected answer or the properties a good answer must have. Draw them from production logs, support tickets and domain experts, then deliberately add the cases that break things: ambiguous questions, missing information, out-of-scope requests and hostile inputs.
Anthropic's evaluation guidance makes two points worth adopting: mirror your real task distribution, edge cases included, and prefer a larger set graded automatically over a small one graded by hand.
Tag every case by slice, such as topic, language, user type or difficulty, so a regression in one slice is not hidden by the average. Version the set with the prompts and code, and add every production failure you fix as a new case so it cannot return unnoticed.
Grade with code first, then rubrics and judges
Use the cheapest grader that can decide each criterion. Code is fast, free and repeatable, so check everything code can check before reaching for a model.
| Grader | Use it for | Watch out for |
|---|---|---|
| Code checks | Valid JSON or schema, required fields, length, banned phrases, citations present, numbers matching the source | They check form, not whether the content is right |
| Reference comparison | Classification labels, extracted fields, short answers with one correct value | Penalises correct answers worded differently |
| Rubric-based LLM judge | Faithfulness, tone, completeness and policy compliance, one criterion at a time | Bias and drift; needs calibration against people |
| Pairwise LLM judge | Choosing between two prompts or models on open-ended answers | Position and length bias; run both orders |
| Human review | Calibrating judges, high-stakes slices, new kinds of failure | Slow and costly; reviewers need written guidelines too |
Using an LLM as a judge without fooling yourself
LLM judges make evaluation scale, and they are biased in known ways. Zheng et al. found a strong judge agreed with human preferences more than 80% of the time, close to the rate at which people agree with each other, but also found biases towards an answer's position, towards longer answers and towards outputs like its own.
Five habits keep a judge honest:
- Ask one yes-or-no question per criterion rather than for a score out of ten; binary verdicts are easier to calibrate and act on.
- Give the judge the rubric, the source context and, where you have one, a reference answer.
- In pairwise comparisons, run both orders and count only consistent verdicts.
- Use a different model from the one that wrote the answer where you can.
- Hand-label a sample, measure how often the judge agrees with it, and recalibrate whenever the judge model or its prompt changes.
Regression testing in CI
Treat prompts, model versions, retrieval settings and tool definitions as code: a change to any of them can break behaviour without a single failing unit test. Run the evaluation automatically on every change that touches them.
- Keep the golden set, graders and judge prompts in your repository, not only in a hosted tool. As of November 2026, OpenAI's hosted Evals platform is read-only and scheduled to shut down on 30 November 2026.
- Pin model versions in configuration, so a provider update is a change you test rather than a surprise.
- Make code checks hard gates: a schema failure or a missing citation fails the build.
- Compare judge scores with the last accepted baseline for each slice, and fail on drops larger than the run-to-run noise you have measured.
- Run each case more than once where outputs vary, and report a pass rate rather than a single result.
- Record cost and latency per case beside quality, so a prompt that is slightly better and much dearer is a visible trade.
- Run a fast subset on every commit and the full set before each release.
Online signals after release
Offline evaluation predicts; production tells you. Users phrase things your golden set never imagined, documents in the index change and providers update models, so evaluation has to continue on live traffic.
Chapter 14 of From Model to Product recommends tracking how often the language model is called, how long it takes, how often responses are empty or invalid and how many tokens it uses. LLM APIs in production covers the reliability side; for quality, add these signals:
- Explicit feedback, such as ratings and reported answers, read as a sample rather than a score, since few users give it.
- Implicit feedback: rephrased questions, retries, abandoned sessions, hand-offs to a person, and whether users accept or edit what was generated.
- The offline judge run on a sample of live traffic with the same rubric, so the two numbers are comparable.
- People reading transcripts regularly. In the AICON Belfast talk on Tellme AI, where Dr. Dulan Dias is co-founder and CTO, reading transcripts is the practical answer to there being no clean metric for a good answer.
- Every confirmed failure fed back into the golden set.
What to measure for RAG and for agents
Retrieval-augmented systems and agents need measures the final answer alone cannot give, because the failure often happens before the output is written.
RAGAS offers reference-free versions of the retrieval measures, and τ-bench judged agents on the final database state against an annotated goal. Reducing LLM hallucinations covers what to do when faithfulness falls, and what is agentic AI explains why agents need repeated runs.
| System | Measure | Why it matters |
|---|---|---|
| RAG | Context recall: did the passages needed come back in the top results? | If retrieval misses, no prompt can fix the answer |
| RAG | Faithfulness: is every claim supported by the retrieved context? | Separates generation errors from retrieval errors |
| RAG | Answer relevance and citation accuracy | A faithful answer can still miss the question or cite the wrong passage |
| RAG | Correct refusal when the context lacks the answer | Rewards saying so instead of filling the gap |
| Agents | Task success judged on the end state, such as the record or file after the run | A transcript can look plausible while the outcome is wrong |
| Agents | pass^k across repeated runs of the same task | One success overstates reliability |
| Agents | Tool choice and argument accuracy at each step | Shows where in the loop the agent goes wrong |
| Agents | Steps, cost and time per completed task, and attempted policy breaches | Catches loops, runaway spend and unsafe actions |
Frequently asked questions
How many test cases do you need to evaluate an LLM app?
Fewer than you think to start, more than you think to finish. A few dozen real cases covering your main slices are enough to catch obvious regressions and begin error analysis. Grow the set from production failures and new edge cases. To detect small quality differences between two prompts or models you need many more, because with a small set the difference disappears into run-to-run noise.
Can you trust an LLM to grade another LLM's output?
Within limits, and only after checking. Research on LLM judges found strong models agree with human preferences about as often as people agree with each other, but also found position, length and self-preference biases. Trust a judge on a criterion once it agrees with a hand-labelled sample at a rate you accept, re-check it when the judge model or prompt changes, and keep people reviewing high-stakes slices.
What is the difference between LLM evals and benchmarks?
A benchmark measures a model's general capability on a fixed public task and is useful for shortlisting models. An eval measures your application, with your prompts, retrieval and tools, on inputs like your users' real traffic. A model that ranks higher on a benchmark can do worse in your product, so treat benchmark results as a reason to test a model, not as the test itself.
How do you evaluate an LLM app when there is no single right answer?
Grade properties instead of matching text. Write down what a good answer must do, such as stay supported by the source, cover the key point, follow the format and stay in scope, and check each separately with code or a calibrated judge. To choose between versions, compare them pairwise in both orders. Reference-free measures such as faithfulness to retrieved context need no gold answer at all.
How often should you rerun LLM evaluations?
On every change to prompts, model versions, retrieval settings or tools, and on a schedule even when nothing has changed. Hosted models are updated and retired by their providers, documents in a retrieval index change, and user traffic drifts. A weekly run of the full set against the production configuration catches changes no commit triggered, and a fast subset on each commit keeps everyday work safe.
Do you need an evaluation platform to start?
No. A spreadsheet of cases, a script that runs them and a few code checks are enough to begin, and they keep the eval set in your own repository. Platforms help with dashboards, tracing and team review as volume grows, but choose one that exports your data, because hosted services can be withdrawn: OpenAI's own Evals platform is scheduled to shut down on 30 November 2026.
How do you evaluate multi-turn conversations?
Replay scripted conversations or use a simulated user, then grade both the turns and the outcome. Per-turn checks catch format and policy breaches; outcome checks ask whether the user's goal was met, such as a booking changed correctly. τ-bench used a language model to play the user and judged agents on the final state of the database, a design worth copying for any assistant that takes actions.
Sources
- Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv 2306.05685)
- Anthropic: Create strong empirical evaluations
- Shankar et al., Who Validates the Validators? (arXiv 2404.12272)
- Es et al., Ragas: Automated Evaluation of Retrieval Augmented Generation (arXiv 2309.15217)
- Yao et al., τ-bench (arXiv 2406.12045)
- OpenAI API deprecations
About the author
Dr. Dulan Dias is co-founder and Chief Technology Officer of Tellme AI in Belfast; founder of EchonLabs, grown from 2017 to more than 60 people before its acquisition in 2026; holder of a PhD in AI and data science from Queen’s University Belfast; and the author of From Model to Product.