What is the difference between training and inference in an LLM?
Training adjusts the model's weights by learning from data and costs enormous compute once. Inference uses the finished, frozen weights to answer a prompt and happens on every request. A chat message never changes the weights.
Explain it simply
Training is a student studying for months before the exam. Inference is the student writing the exam. Studying changes what is in the head; writing the exam does not, at least not permanently.
During training the model reads huge amounts of text, predicts the next token, measures its error and nudges billions of numbers (weights) to do better. This runs on large GPU clusters for a long time. During inference those weights are fixed. Your prompt goes in, tokens come out, and nothing is learned.
This is why telling a chatbot "remember that my name is Asha" works only inside the current conversation: the name sits in the prompt text, not in the weights. To change what the model knows permanently you must fine-tune or retrain it.
Example
Training: data -> compute loss -> update weights (repeated many times)
Inference: prompt -> frozen weights -> generated tokensWhat to say in the interview
"Training updates the weights from data and is expensive and infrequent. Inference runs the frozen model per request and its cost scales with usage. A user's chat does not update weights, so new knowledge comes in through the prompt, retrieval, or a separate fine-tuning job."
Common mistakes
- Thinking the model learns from every user conversation in real time.