How do you measure the quality of retrieval in a RAG system?
Use a set of questions with known correct passages and compute recall at k, the share of questions where a correct chunk appears in the top k, and MRR, which rewards ranking the correct chunk higher. Check retrieval separately from the generated answer.
Explain it simply
If your RAG bot gives a wrong answer, either the right passage never reached the model or the model misused it. You can only tell which if you measure retrieval on its own.
Prepare 50 or more questions and mark which chunk or document contains each answer. Then compute:
Recall at k: for how many questions was a correct chunk within the top k results? If it was in the top 5 for 40 of 50 questions, recall at 5 is 80%.
MRR (mean reciprocal rank): for each question take 1 divided by the rank of the first correct chunk (rank 1 gives 1, rank 2 gives 0.5, rank 4 gives 0.25), then average. It rewards putting the right chunk near the top.
Compare chunk sizes, embedding models, hybrid search and reranking using these numbers.
Example
Three questions, correct chunk found at ranks 1, 3 and not in top 5:
reciprocal ranks: 1/1 = 1.000, 1/3 = 0.333, 0
MRR = (1.000 + 0.333 + 0) / 3 = 0.444
Recall at 5 = 2 of 3 = 66.7%What to say in the interview
"I evaluate retrieval separately with labelled queries, using recall at k for coverage and MRR for ranking quality. That tells me whether to improve chunking, embeddings, hybrid search or reranking before touching the generator."