How to Evaluate RAG Retrieval Before Tuning the Model

A retrieval-augmented generation system can produce a polished answer from the wrong documents. Upgrading the language model won't fix a missing policy page, a stale product guide, or a search filter that excludes the right evidence. RAG retrieval evaluation starts one step earlier: did the system find the material a careful person would need?

This guide shows how to build a small, repeatable evaluation for a company knowledge assistant. The examples are fictional test cases, not measured results from a live product. You can adapt the workflow to support articles, internal documentation, or a customer-facing help centre without starting with an expensive benchmark platform.

Separate retrieval quality from answer quality

Think of the pipeline as three separate decisions: which documents enter the index, which passages search returns, and what the model says about them. A failure at any stage can look like a bad answer, but the fixes aren't interchangeable. A correct passage with a wrong conclusion points toward generation. An absent passage points toward ingestion or retrieval.

The OWASP RAG Security Cheat Sheet also treats document provenance, access boundaries, and retrieved content as security concerns. For your evaluation, record both relevance and permission eligibility. A result is not useful evidence if that user shouldn't be able to read it.

Build a small evidence set people can defend

Start with roughly 30 to 50 questions covering actual tasks. That's a suggested pilot size, not a universal statistical threshold. Include simple lookups, multi-document questions, ambiguous wording, and questions whose answer isn't in your corpus. Ask the document owner to identify acceptable evidence before running the assistant.

For each case, store a question, the user's role, expected document identifiers, required facts, and an evaluation date. Don't store private document text in a public debugging tool. A sanitised record can look like this:

{
  "case_id": "refund-window-01",
  "question": "When can a customer request a refund?",
  "role": "support",
  "required_document_ids": ["refund-policy-current"],
  "must_abstain": false
}

Use the JSON Formatter to inspect the structure of a synthetic evaluation fixture. Formatting makes the records easier to review; it doesn't verify whether the expected evidence is correct. Keep that decision with someone who understands the policy.

Measure the returned evidence, not its confidence

For each question, save the top retrieved passages and their document versions. Recall at a chosen cut-off asks whether the required evidence appears among those results. Precision asks how many returned results are relevant. For multi-document tasks, count whether all necessary sources were retrieved, not just whether one plausible source appeared.

A simple example makes the difference clear. If a question requires documents A and B, and the top five results contain A but not B, document-level recall is one half for that case. Don't call the case successful merely because the generated answer sounds reasonable. Likewise, a high retrieval score from your search engine isn't a calibrated probability of correctness.

Choose metrics that match the task. A policy lookup may need one authoritative page. An incident summary may need several dated records. Report those groups separately so easy questions don't hide a weak result on the harder work users actually care about.

Run controlled experiments

Freeze the question set, corpus version, and user permissions. Then change one retrieval setting at a time: chunk size, overlap, search strategy, filters, or reranking. Save each run with a configuration identifier so a good result can be reproduced later.

Compare sanitised configurations with JSON Diff. If a run changed the embedding model, chunk boundaries, and filters together, you haven't learned which change helped. Keep a held-out set of questions you don't use for tuning, and examine regressions as carefully as improvements.

Don't use the same model's enthusiastic self-assessment as your only judge. Automated grading can help triage a large result set, but a person should review important disagreements, missing evidence, and unexpected disclosures. Define the scoring rubric before looking at the output.

Turn failures into a repair list

Label each failed case with the earliest broken stage. Was the document missing from ingestion? Did parsing lose a table? Did a date filter select an obsolete page? Did synonyms prevent a match? Did a chunk split the condition away from its exception? Each label suggests a different repair.

For example, a refund answer may depend on both a normal window and a special rule for subscriptions. If those clauses are separated, increasing the number of retrieved passages might help, but preserving section context could be better. Test the proposed repair on the whole set, not only on the question that exposed it.

Include negative cases deliberately. Ask about an unsupported feature, an old policy, and a document restricted to another role. A healthy assistant should acknowledge missing evidence or refuse access rather than manufacture a convenient answer. Measure these behaviours alongside ordinary relevance.

Decide what is good enough to release

Set release criteria by consequence. A low-risk help article search can tolerate different errors from an assistant explaining contractual obligations. Record the acceptable miss rate, the cases requiring human review, and the owner who approves changes. Don't borrow a benchmark score without understanding its task distribution.

Also check latency and retrieval cost. A more accurate search that times out under normal load may not improve the user experience. For spending controls around the broader assistant, see AI agent cost budgets.

Conclusion

Evaluate RAG by checking whether the right, current, permitted evidence reaches the model. A small labelled set, controlled experiments, and honest failure categories will tell you more than a confident demo. Improve retrieval first, then tune the answer layer against evidence you can explain.