RAG testing: how to know your retrieval is actually working

When a RAG answer is wrong, the bug is usually in retrieval, not generation, but most teams only ever look at the final answer. The model gets blamed for a bad answer when it was handed bad context and did its best with it. To test RAG properly you have to pull the two stages apart and measure them separately.

Test retrieval and generation as two systems

A RAG pipeline is: query -> retrieve chunks -> generate answer from chunks. Each stage fails differently:

• Retrieval fails by not surfacing the relevant chunk (or burying it under noise). No model can answer from context it never received.
• Generation fails by ignoring good context, hallucinating beyond it, or not citing.

If you only score the final answer, you can’t tell which one broke, so you tune prompts when the real problem is your chunking or embeddings. Measure both.

Retrieval metrics that matter

Build a small set of queries with the chunk(s) you know should be retrieved, then measure:

• Recall@k, is the relevant chunk in the top k? This is the big one. If recall is low, nothing downstream can save you.
• Precision / noise, how much irrelevant context are you stuffing in? Noise degrades generation and runs up cost.
• “Is the answer even in here?”, before judging an answer, confirm the needed fact was retrieved at all.

def test_retrieval(case):
    chunks = retrieve(case["query"], k=5)
    ids = {c.id for c in chunks}
    # The chunk we know contains the answer must be retrieved.
    assert case["relevant_id"] in ids, f"recall miss: {case['query']}"

A retrieval test is deterministic and cheap, it doesn’t even call the LLM. Run it on every change to chunking, embeddings, or the index.

Grounding and citation: did it use the context or make it up?

Once retrieval is solid, score the generation for groundedness, every claim in the answer should be supported by the retrieved chunks, and citation accuracy. The eval rubric is the place this judgment lives:

def test_grounded(case):
    chunks = retrieve(case["query"], k=5)
    answer = generate(case["query"], chunks)
    result = score(rubric="rag_grounding_v1", output=answer, context=chunks)
    assert result.grounded >= 0.9, result.explanation
    # And it should refuse rather than guess when context is missing.
    if case["unanswerable"]:
        assert is_refusal(answer)

That last check matters more than it looks: a RAG system that confidently answers when the context doesn’t contain the answer is the most dangerous failure mode, because it’s invisible until a user is misled.

Catching silent retrieval rot

RAG quality decays as your corpus changes, new documents shift what gets retrieved, re-embedding moves rankings, a chunking tweak quietly drops recall. None of this touches your prompt, so prompt-level tests won’t catch it. Guard against it by running the retrieval set in CI and alerting when recall drops against the baseline, the same way you’d gate a prompt regression:

$ eval run --suite rag --baseline main
  recall@5 (main) ...... 0.94
  recall@5 (this PR) ... 0.81
FAIL  retrieval regressed - 6 queries lost their relevant chunk

Wire it into the release gate

Retrieval tests (deterministic) on every PR; grounding evals (scored) on the AI surface; both blocking. Add a golden query whenever a real RAG failure escapes. That’s the same release-gate discipline as the rest of LLM testing, retrieval just happens to be the half everyone forgets to measure.

---

If your RAG feature works in the demo but you can’t say why it sometimes answers wrong, that gap is exactly what a QA Foundation Sprint closes, separating retrieval from generation, putting numbers on both, and gating them in CI.