What do temperature and top-p control in an LLM?

At every step an LLM gives a score to each possible next token. Imagine a canteen menu where each dish has a popularity score. You could always order the most popular dish, or you could pick with some randomness so you sometimes try something else.

• Temperature changes how strongly you favour the popular dishes. Low temperature: you almost always pick the top dish. High temperature: the less popular dishes get a real chance.
• Top-p (also called nucleus sampling) first throws away the unpopular tail of the menu. With top-p of 0.9, you keep only the most popular dishes until their shares add up to 90%, and pick only among those.

Neither setting makes the model smarter or adds knowledge. They only change how it chooses among options it already considered likely.

The model's raw scores (logits) are turned into probabilities with a function called softmax. Temperature divides the scores before softmax:

function softmax(logits, temperature) {
  const exps = logits.map((x) => Math.exp(x / temperature));
  const sum = exps.reduce((a, b) => a + b, 0);
  return exps.map((x) => (x / sum).toFixed(2));
}

const logits = [2, 1, 0]; // scores for three candidate tokens

console.log(softmax(logits, 1)); // Output: ["0.67","0.24","0.09"]
console.log(softmax(logits, 0.5)); // Output: ["0.87","0.12","0.02"]
console.log(softmax(logits, 2)); // Output: ["0.51","0.31","0.19"]

At temperature 0.5 the top token gets 87% of the probability, so output is very predictable. At temperature 2 the choice is much more spread out.

With top-p = 0.9 on the temperature-1 numbers, the model keeps the first two tokens (0.67 + 0.24 = 0.91, which crosses 0.9), drops the third, rescales the two survivors, and samples between them.

1. The model outputs one score per token in its vocabulary.
2. Scores are divided by the temperature, then passed through softmax to get probabilities.
3. If top-p is set, tokens are sorted by probability and only the smallest group that reaches p is kept.
4. One token is sampled from what remains.

A temperature of 0 is usually treated as "always pick the highest-probability token" (greedy decoding). Even then, outputs may not be perfectly identical across runs on some systems because of how the computations are batched and rounded.

• Extracting fields from an invoice or classifying support tickets: low temperature, because you want consistent answers.
• Brainstorming taglines for a college fest: higher temperature, for variety.
• Many teams change either temperature or top-p, not both, so the effect of each change is easy to understand.

• Thinking temperature 0 removes hallucinations. It makes output repeatable, not correct.
• Setting temperature very high and getting incoherent text, then blaming the model.
• Confusing top-p with top-k. Top-k keeps a fixed number of tokens; top-p keeps a variable number based on cumulative probability.

• What is the difference between top-p and top-k sampling?
• Why might you still see different outputs at temperature 0?
• Which setting would you use for a JSON extraction task, and why?