What is reranking in RAG, and why add it after retrieval?
Reranking takes the top candidates from fast retrieval, say 50, and rescores each with a more accurate but slower model that reads the query and passage together. You then send only the best few to the LLM, improving relevance.
Explain it simply
Think of campus placements. The first round is a quick resume screen that picks 50 from 2,000 applicants. The second round is a detailed interview of those 50 and picks the top 5. You cannot interview all 2,000, and the quick screen alone is not accurate enough.
Vector retrieval is the quick screen: each document is embedded in advance, so search is fast but coarse. A reranker, often a cross-encoder, reads the query and one candidate together and outputs a relevance score. It sees interactions between the words, so it is more accurate, but it must run per candidate at query time, which costs latency.
Typical flow: retrieve around 20 to 100 candidates, rerank, keep the top 3 to 8 for the prompt. Fewer, better chunks give the LLM a cleaner context.
Example
retrieve top 50 by embedding similarity (fast, approximate)
rerank 50 with a cross-encoder (slower, precise)
keep top 5 -> put in promptWhat to say in the interview
"Reranking is a second-stage precision step: broad cheap retrieval for recall, then a cross-encoder rescoring so the final context is more relevant. It adds latency, so I choose the candidate count carefully and measure the quality gain."