How to evaluate an LLM application before you launch it

Why a good demo proves little

A language model application almost always demonstrates well. Someone tries five questions they chose, the answers read fluently, and the room is convinced. But fluent and correct look the same on the page, and the questions real users ask are stranger than the ones a builder thinks of. Model output also varies from run to run, so one good answer says little about the next.

Evaluation replaces impressions with a repeatable measurement. You hold a fixed set of tasks, a written definition of a good result and a way of scoring, and you run them every time something changes. This article describes the method for an application that answers from your documents.

Plan the work at the start: the test set is easiest to build while you gather requirements.

Build a test set from real tasks

Take the tasks from where the work happens: support tickets, emails to the help desk, search logs, the questions new staff ask. Handle them under your privacy rules and remove personal information. Invented questions are tidier than real ones and will flatter the system.

Cover the range. Include the common questions, then deliberately add the hard ones: vague wording, spelling mistakes, two questions in one, questions that need two documents, questions about something that changed recently, and questions your documents cannot answer. Add a few attempts to make the assistant misbehave, such as requests to ignore its instructions or reveal restricted material.

Fifty to a hundred cases is enough to begin. Keep a portion aside that nobody tunes against, so you have an honest check before launch. Store the set in version control, with an owner.

• Real tasks from real channels
• Common cases and hard cases
• Unanswerable questions
• Misuse attempts
• A held-back portion for the final check

Define what a good answer is

For each case, have someone who knows the subject write the reference: the correct answer, the document and section it comes from, and anything the answer must contain or must never contain. This is slow work, and it is the most valuable time spent in the project, because it forces the business to agree what correct means.

Then write the scoring criteria as separate yes or no checks. A single score out of ten hides what went wrong, and two reviewers will rarely agree on it.

• Correct: the facts match the reference
• Grounded: every claim is supported by the retrieved text
• Complete: nothing the user needs is missing
• Honest: it declines when the sources are silent
• Fit for use: right length, tone and format

Separate retrieval quality from answer quality

A document-grounded application has two stages, and they fail differently. Retrieval finds passages. Generation writes the answer from them. If you score only the final answer, you cannot tell which stage to fix.

Score retrieval first. For each case, check that the passage holding the answer was returned, and how near the top. If it was missing, no prompt change will help: the fix lies in how documents are split, indexed, tagged or searched, or in the documents themselves. Then score generation on the cases where retrieval succeeded. Failures here are the model ignoring the passage, adding facts from outside it, merging two conflicting passages, or missing a condition in the text. Those are fixed through the prompt, the choice of model or checks on the output.

Report the two numbers side by side. Early failures usually trace back to retrieval and to the sources. That is the more fixable half, because it is ordinary document and search work.

Human review and automated scoring

Review by subject experts is the reference standard. It is also slow, costly and inconsistent unless you manage it. Give reviewers the yes or no criteria, have two people score the same sample, and discuss where they disagree. The disagreements usually expose an unclear criterion.

Automated scoring lets you run the whole set on every change. Simple checks cost almost nothing: did the answer cite a source, is the cited passage the expected one, does it contain the required figure, did it decline when it should. For judgement criteria such as groundedness, a second model can act as a grader, given the question, the passages, the answer and one criterion at a time.

Treat a model grader as an instrument that needs calibration. Compare its verdicts with your experts on a sample, adjust its instructions until they agree closely enough for your purpose, and recheck from time to time. Use automation for breadth and speed. Use people for the launch decision, for high-stakes topics and for a regular sample.

Regression checks when anything changes

Three things change in a live application: the prompts, the model and the documents. Each can break answers that used to be right. A prompt edit that fixes one case often damages another. A model provider updates or retires a version. A policy document is replaced, and the old reference answer is now wrong.

Run the full test set before any of these changes goes live, and compare it with the last accepted run case by case. An unchanged average can hide five fixes and five new failures. Decide in advance what blocks a release, such as any new failure on a high-stakes case.

• A full run before every prompt, model or document change
• Case-by-case comparison with the last accepted run
• Written rules for what blocks a release
• Pinned model versions, upgraded on your schedule
• Reference answers updated along with the documents

Monitoring after launch

The test set is a sample of what you expected. Production shows you what you did not expect. Log every question, the passages retrieved, the answer and any user feedback, within your privacy and compliance requirements. Give users a one-click way to flag a bad answer.

Watch a small set of signals: flagged answers, questions where nothing relevant was retrieved, refusals, handovers to a person, response time and cost. Have an expert review a random sample on a schedule, since users flag only some of the errors they meet. Every confirmed failure goes into the test set with its reference answer, so the set grows towards real use.

We build company brains, which are multi-purpose assistants trained on a company's own data. In systems like these the documents change far more often than the model does, so a monitoring finding often ends as a document fix.

What to do next

Collect fifty real questions this week and ask one subject expert to write the reference answer and source for each. Run your current system against them, and score retrieval and answers separately. Whatever the result, you now have a baseline. For a second opinion on the method, talk to our AI consulting team.