Temperature, top-k and top-p, explained visually

A language model doesn't choose a word. It gives every word in its vocabulary a probability, and something then has to pick one. These settings change how that happens. The starting odds below are written by hand, and the reshaping and the sampling are real.

What each setting does

• Temperature rescales the model's scores before they become probabilities. Below 1, likely words get likelier. Above 1, the odds flatten and unlikely words get real chances.
• Top-k keeps only the k likeliest words and drops the rest.
• Top-p keeps the smallest set of top words whose probabilities add up to p, so the list is short when the model is sure and long when it isn't.

Here they apply in that order, and whatever is left is rescaled to add up to 100%.

Choosing values

Use a low temperature when you want the same dependable answer, as when extracting data. Use a higher one when you want variety, as when brainstorming. Very high values mostly add nonsense rather than creativity. Providers offer different settings and ranges, and it's usually enough to change one at a time and look at the results.

Where the probabilities come from

Softmax turns the model's raw scores into probabilities, and temperature works inside it.