How do you choose chunk size in a RAG system?
Small chunks give precise matches but may lose surrounding context. Large chunks keep context but dilute the embedding and waste prompt space. Pick a starting size from the document type, then tune it by measuring retrieval quality on real questions.
Explain it simply
Think of cutting a cake for guests. Slices that are too thin fall apart. Slices too thick are more than anyone wants. Chunk size has the same sweet spot.
Tiny chunks, say one sentence, match narrow questions well but may lack the "because" or "except" that makes the answer correct. Huge chunks hold complete context but embed several topics together, match less precisely and fill the prompt with filler.
A common approach is to start with a few hundred tokens and respect natural boundaries such as headings, paragraphs and list items. FAQs may suit one chunk per question. Legal clauses may need larger chunks. Then test: build a set of real questions with known source passages and compare recall at top-k for a few sizes. Decide using the numbers, not guesses.
Example
Sizes tested on 50 real questions (hit = right passage in top 4):
100 tokens -> 62% hit
300 tokens -> 78% hit
800 tokens -> 71% hit
Choose around 300 for this corpus.These figures are illustrative; your corpus will differ.
What to say in the interview
"Chunk size trades precision against context. I start from document structure, then evaluate a few sizes with recall at k on real queries and pick the best, while watching prompt cost."