Evaluate retrieval before trusting a RAG answer
A failure-oriented approach to testing retrieval, permissions, citations, answer support, and stale knowledge in an AI knowledge system.
SHORT ANSWER
Evaluate a RAG system in layers: source coverage and permissions, retrieval relevance, context sufficiency, citation correctness, answer support, refusal behavior, and freshness. Testing only the final prose hides where failures originate.
Start with the source boundary
List which systems are authoritative, who may access each source, how quickly updates must appear, and what should never enter model context. Retrieval quality cannot compensate for missing, stale, duplicated, or incorrectly permissioned sources.
Use questions that expose failure
Build questions from real work, then add ambiguous, outdated, multi-source, unanswerable, and permission-sensitive cases. For each question, record the expected sources and what a responsible refusal should say.
- Answerable from one clear source
- Requires synthesis across sources
- Source exists but user lacks access
- Information is outdated or contradictory
- No approved source contains the answer
Score retrieval separately
Inspect whether the required source appears, whether irrelevant context crowds it out, and whether metadata filters and permissions behave correctly. If retrieval fails, judging the final generation tells the team little about the remedy.
Verify every citation relationship
A citation link is not proof that the cited passage supports the statement. Check that the passage exists, is accessible to the user, supports the specific claim, and is not contradicted by a more authoritative or recent source.
Design refusal and escalation
Useful systems state when evidence is insufficient, show the search boundary, and route unresolved questions. A fluent answer to an unanswerable question is a production defect, even when it sounds helpful.