Why can an LLM give different answers to the same question?
Most chat settings sample the next token from a probability distribution instead of always taking the top one, so small random choices early on lead to different wording or even different conclusions. Lowering temperature reduces this variation.
Explain it simply
Imagine a cricket commentator who has five good ways to open a sentence and picks one at random each time. Nobody is confused, but no two commentaries match word for word.
An LLM does the same. At each step several tokens are reasonable, and the system samples among them. Once one word differs, every later word is conditioned on it, so the answers drift apart. For a factual question the drift is small. For an open one, such as "suggest a project idea", it is large.
Other causes exist too: the product may change the system prompt, a different model version may be served, and on busy servers tiny numeric differences can flip a close choice even at temperature zero.
Example
Prompt: Give me a one-line slogan for a chai stall.
Run 1: "Sip the moment, one kulhad at a time."
Run 2: "Chai that feels like home."
Run 3: "Where every break tastes better."All three are valid. Same prompt, different samples.
What to say in the interview
"Generation is usually stochastic: the model samples from a distribution rather than picking the single most likely token. Differences compound because each token depends on the previous ones. For repeatable output I would lower temperature, fix a seed if the API supports it, and still not promise bit-exact results."
Common mistakes
- Believing temperature 0 guarantees identical output on every call. It makes it far more stable, not perfectly so.