Build the evaluation set before you tune your RAG pipeline

A retrieval-augmented generation system has a lot of dials. Chunk size and overlap, the embedding model, how many passages to retrieve, whether to rerank them, how the prompt frames the context. Every one of them changes the answers, and every one of them is usually tuned the same way: someone asks a handful of questions, reads the output, and decides it looks better than yesterday.

That method has a ceiling, and teams hit it quickly. Improvements on the questions you happened to try quietly break the ones you didn't. Two people read the same answer and disagree about whether it is right. The pipeline drifts toward whatever the last reviewer cared about. The fix is not a better model. It is a measuring instrument, built before the tuning starts.

What an evaluation set actually is

It is a list of real questions, each paired with what a correct answer must contain and which documents it should come from. Not a benchmark downloaded from the internet, and not questions invented by the engineer who built the pipeline. The questions should come from the people who will use the system, in their words, including the vague and badly phrased ones, because that is what production traffic looks like.

• The question, verbatim as a user would ask it
• The key facts a correct answer must include, written as a short checklist rather than a model answer
• The source documents or sections that contain those facts
• A note when the honest answer is “the documents don't say”

That last item matters more than it looks. A system that answers confidently when the material is silent is more dangerous than one that answers badly, and you only catch it if some of your test questions have no answer in the corpus.

How big it needs to be

Smaller than people expect. Fifty to a hundred well-chosen questions are enough to detect the difference between a good configuration and a bad one, and they can be assembled in a few days. What matters is coverage: every document type, every user group, the common questions and a deliberate sample of the awkward ones. A thousand near-duplicate questions about the same policy page measure one thing a thousand times.

Measure retrieval and generation separately

When an answer is wrong, there are two very different reasons. Either the right passage was never retrieved, or it was retrieved and the model ignored or misread it. Mixing them into one score hides which half of the pipeline to fix. Because each test question records its source documents, you can check retrieval on its own: did the expected passage appear in the top results at all?

• Retrieval hit rate: how often the expected source appears in the retrieved set
• Answer completeness: how many of the required facts the answer contains
• Groundedness: whether every claim in the answer is supported by a retrieved passage
• Refusal accuracy: whether the system says it doesn't know when the corpus is silent

If retrieval hit rate is low, no amount of prompt engineering will help; work on chunking, metadata and search. If retrieval is good and answers are still wrong, the problem is downstream, in the prompt, the context ordering, or the model itself.

Automate the scoring, keep a human in the loop

Retrieval metrics are mechanical and can run on every change. Answer quality is harder. A second model can check answers against the fact checklist reasonably well, which makes it cheap to run the whole set after each experiment. But automated grading drifts too, so a person should spot-check a sample of the grades every so often, and re-label anything the grader and the human disagree on.

Treat the set like code

Store the evaluation set in version control next to the pipeline configuration. Run it in CI when the prompt, the chunking or the model changes, and record the scores with the commit. When someone proposes a change, the conversation becomes “retrieval went up four points and completeness dropped one” instead of “it feels better”.

Add to it whenever production produces a bad answer. Every complaint from a user is a free test case, and a set that grows from real failures is the closest thing to a guarantee that the same mistake will not ship twice.

Where to start on Monday

Collect forty real questions from the people who will use the system. Write down the facts each answer needs and where they live. Run the current pipeline against them and score it, even roughly. That baseline will be humbling, and it will be the first number anyone on the project can honestly argue with.