When would you use prompting, RAG or fine-tuning to adapt an LLM?
Start with prompting because it is cheapest and fastest to change. Add RAG when the model needs knowledge it does not have, such as private or changing data. Use fine-tuning when you need consistent behaviour, style or format that prompting cannot reliably achieve, not mainly to add facts.
Explain it simply
Think about a new hire at a bank's customer-support desk.
- Prompting is giving clear instructions each morning: "Be polite, reply in short sentences, escalate fraud cases."
- RAG is giving them access to the policy binder, so they look up the exact rule before answering.
- Fine-tuning is a training programme. After weeks of practice on thousands of real conversations, they naturally reply in the bank's style without needing to be reminded.
Instructions are quick to change. The binder handles knowledge that changes often. Training changes habits, but it takes effort, and if the rules change next month, the training does not update itself.
Example
| Need | Best first choice | Why |
|---|---|---|
| Answer from this year's HR policy PDF | RAG | Private, changes every year, needs citations |
| Reply always in a fixed JSON format | Prompting with structured output | Usually enough, and quick to change |
| Classify tickets into 40 internal categories with high consistency | Fine-tuning (after trying prompts) | Many labelled examples exist, and prompts alone are inconsistent |
| Write in the company's exact brand voice at scale | Fine-tuning, or few-shot prompting first | Style is behaviour, not knowledge |
| Know today's exchange rate | A tool or API call | It changes too fast for any of the three |
How it works
Prompting changes only the input. No training, no new infrastructure, and you can update it in minutes. Its limits are the context window and the fact that long, complex prompts can still be followed inconsistently.
RAG adds a retrieval system in front of the model. Knowledge lives in documents you can update, delete or restrict per user, and answers can cite sources. Costs are an indexing pipeline, a vector store and extra tokens per request. Quality depends on retrieval.
Fine-tuning continues training the model on your own examples, usually input-output pairs. It changes the weights, so the behaviour is built in and prompts can be shorter. Parameter-efficient methods such as LoRA train only a small set of extra weights, which makes it cheaper. Costs are collecting good labelled data, running training, evaluating, and repeating whenever requirements change. Fine-tuning is a weak way to add facts: the model may blend them up or hallucinate around them, and you cannot easily update or remove a single fact or show a source.
The usual order is: prompt first, measure, add RAG if knowledge is missing, and fine-tune only when you have a clear gap that prompting and RAG did not close and enough good examples to train on. They also combine well, for example a fine-tuned model that follows your format, used inside a RAG pipeline that supplies current facts.
Real-world use
- A college helpdesk bot: prompting plus RAG over notices and the syllabus.
- A high-volume ticket router: fine-tuned small model, because it is cheaper per request and more consistent.
- An internal coding assistant: RAG over the codebase, since code changes daily.
Common mistakes
- Fine-tuning to "teach the model our documents". RAG is usually the better tool for knowledge.
- Skipping evaluation. Without a test set you cannot tell whether a change helped.
- Fine-tuning on a few hundred messy examples and expecting big gains. Data quality matters more than the method.
- Treating the choice as either-or. Real systems often use all three.
What to say in the interview
"I'd start with prompting, because it's the cheapest and easiest to iterate, and build an evaluation set first. If the model lacks knowledge, especially private or frequently changing data, I'd add RAG, which also gives citations and access control. I'd fine-tune when I need consistent behaviour, format or style, or a smaller cheaper model for a narrow task, and I have enough quality examples. Fine-tuning changes how the model behaves; RAG changes what it knows at answer time. They often work together."
Follow-up questions
- Why is fine-tuning a poor way to add new facts?
- What is LoRA, and why is it popular for fine-tuning?
- How would you build an evaluation set before choosing an approach?
Go deeper
Read these once the basics above feel easy.
Fine-tuning has hidden costs. Training on a narrow task can reduce the model's general ability or safety behaviour, so you must re-test broadly, not only on the target task. Each fine-tuned model must be versioned, hosted and retrained when the base model is updated. Data leakage is another risk: anything in the training set, including personal data, can be reproduced by the model and cannot be selectively deleted later, whereas a document in a RAG index can simply be removed. Distillation is a related technique: a large model generates high-quality outputs for your task, and a smaller model is fine-tuned on them to get similar quality at lower cost.