How do you decide how many chunks (top-k) to retrieve in RAG?
Too few chunks risk missing the answer. Too many add noise, raise cost and can bury the useful passage. Choose k by measuring recall and answer quality on test questions, and often retrieve many, rerank, and pass only a handful to the model.
Explain it simply
Imagine asking a librarian for books on a topic. Handing you one book may miss the right chapter. Handing you fifty buries you.
In RAG, k is the number of chunks passed to the model. A small k is cheap and focused but fails when the answer needs information from several places, or when the best chunk is ranked sixth. A large k raises recall but adds tokens, slows the response and gives the model more irrelevant text to be distracted by.
A practical recipe: measure recall at several k values on a labelled question set, then choose the smallest k that reaches your target. Improve further by retrieving a wider set, such as 30, reranking, and sending only the best 4 to 6. Questions that need comparing several documents may need a larger k than simple lookups.
Example
k = 3 -> recall 70%, short prompt
k = 6 -> recall 88%
k = 12 -> recall 91%, much longer prompt, small gain
Choose k = 6.Figures are illustrative.
What to say in the interview
"I tune k empirically: pick the smallest value that gives acceptable recall, and use retrieve-wide, rerank, then keep-few to balance recall against noise and token cost."