What is retrieval-augmented generation (RAG), and when would you use it?
RAG first retrieves relevant passages from your own data, then gives them to the LLM along with the question so it answers from that material. Use it when answers must come from private, large or frequently changing information the model was not trained on.
Explain it simply
Think of an open-book exam. A student who studied well but has no book must answer from memory and may misremember details. A student allowed to bring the textbook can look up the right page first and then write the answer. RAG turns the LLM into the second student.
An LLM only knows what was in its training data, and that data has a cut-off date and does not include your company's internal documents. RAG fixes this without retraining the model. Before answering, the system searches your documents, picks the most relevant passages, and pastes them into the prompt with an instruction like "Answer using only the context below."
Example
A bank wants a support chatbot that answers questions about its own account rules.
User question:
"What is the minimum balance for a savings account in a metro branch?"
1. Retrieve: search the bank's policy documents
-> finds chunk from "Savings Account Rules", section 3.2
2. Augment: build the prompt
System: Answer only from the context. If the answer is not there, say you don't know.
Context: [text of section 3.2]
Question: What is the minimum balance for a savings account in a metro branch?
3. Generate: the LLM answers using section 3.2 and cites itWhen the bank changes its policy, it updates the document and re-indexes it. The model itself does not change.
How it works
A RAG system has two pipelines.
Indexing (done ahead of time)
- Load documents such as PDFs, web pages or help articles.
- Split them into chunks, often a few hundred tokens each, sometimes with a little overlap.
- Convert each chunk to an embedding and store it in a vector index, along with the original text and metadata like source and date.
Answering (done per question)
- Embed the user's question.
- Retrieve the top few most similar chunks. Many systems combine this with keyword search, because exact terms like product codes are matched better by keywords.
- Optionally re-rank the results with a more precise model.
- Put the chunks and the question into the prompt and let the LLM generate the answer, ideally with citations.
Real-world use
- Internal helpdesks that answer HR or IT policy questions.
- Customer support bots grounded in product manuals and FAQs.
- Search over a college's notices, syllabus and exam rules.
- Coding assistants that look up the relevant files of a large codebase.
Common mistakes
- Blaming the LLM when the real problem is retrieval. If the right chunk is not retrieved, the answer cannot be right. Test retrieval quality separately.
- Bad chunking, such as splitting a table or a rule in half.
- Not telling the model what to do when the context lacks the answer, which invites made-up answers.
- Ignoring access control. A user should only retrieve documents they are allowed to see.
What to say in the interview
"RAG is retrieval-augmented generation. Instead of relying only on what the model memorised in training, we search our own knowledge base for relevant passages, usually with embeddings and a vector index, and put them in the prompt so the model answers from that context. I would use it when the information is private, changes often or is too large to fit in a prompt, and when we need citations. It reduces hallucinations but does not remove them, and the quality depends heavily on chunking and retrieval."
Follow-up questions
- How would you evaluate whether a RAG system is working well?
- Why combine keyword search with vector search?
- What would you do if the retrieved chunks contradict each other?
Go deeper
Read these once the basics above feel easy.
Evaluate the two halves separately. For retrieval, build a small set of real questions with the chunk that should answer each, and measure how often that chunk appears in the top results. For generation, check whether the answer is supported by the retrieved context (faithfulness) and whether it actually answers the question (relevance). Most improvements come from the retrieval side: better chunking, adding metadata filters, hybrid search and re-ranking.