A RAG evaluation dataset is a curated set of questions, expected evidence, acceptable answers, and known failure cases. It gives a team something stable to test when sources, chunking, retrieval, prompts, models, or permissions change.

Start from real business questions

Collect representative requests from the workflow, including common questions, difficult wording, misspellings, ambiguous cases, and questions the system should refuse. Remove unnecessary personal information and record the business context needed to judge the answer fairly.

Store evidence, not only answers

For each case, identify the authoritative source, relevant passage, required qualifiers, and unacceptable claims. An answer can sound plausible while using the wrong document or missing a restriction. Evaluate retrieval relevance and citation quality separately from generated wording.

Test permissions and change

Include cases where two users should see different sources, where a document is stale, and where sources conflict. Run the set after changes to indexing, metadata, access rules, retrieval settings, prompts, or models. The data readiness guide helps improve the source collection, and the metrics guide helps report results.

• Version the dataset and its expected evidence.
• Keep safety and refusal cases alongside routine cases.
• Review failures by root cause, not only by score.
• Preserve a small holdout set for meaningful comparisons.

A useful dataset is deliberately imperfect and regularly refreshed. It should represent the questions the business receives, not only the questions that make a demo look good.