Sampling strategies

Temperature on a longer list

Now try seven possible next words after "It is", from a product catalog. The code prints each word's chance at three temperatures and draws them on one chart.

import numpy as np
import matplotlib.pyplot as plt

words = ["light", "warm", "durable", "packable", "waterproof", "purple", "edible"]
logits = np.array([2.0, 1.6, 1.4, 1.1, 0.9, -1.0, -2.5])      # next word after "It is"

def softmax(z, T=1.0):
    e = np.exp((z - z.max()) / T)
    return e / e.sum()

for T in [0.5, 1.0, 2.0]:
    p = softmax(logits, T)
    print(f"T = {T}: " + "  ".join(f"{w} {q:.2f}" for w, q in zip(words, p)))
    plt.plot(range(len(words)), p, marker="o", label=f"T = {T}")
plt.xticks(range(len(words)), words, rotation=30)
plt.ylabel("probability")
plt.legend()

At $T = 0.5$, "light" alone takes about half the chance, and the top few words crowd out the rest. At $T = 2$, the line flattens. "edible" climbs from almost nothing to about 2.6% (printed as 0.03), so roughly one pick in 40 would call the product edible. That's how a high temperature lets odd words slip in.

The softmax here subtracts the largest score before it does anything else. That keeps the numbers small enough for the computer to handle, and it doesn't change the answer.

Top-k and top-p

Temperature reshapes the whole list, but it never removes anything. Even a silly word keeps a tiny chance. The next two methods cut the unlikely words, called the tail, out of the list. Then they share the chance among the words that are left, so it adds up to 1 again. This step is called renormalizing.

• Top-k: keep only the $k$ most likely tokens. Here $k$ is a whole number you choose, such as 3 or 50.
• Top-p, also called nucleus sampling: sort the tokens from most to least likely. Then keep adding tokens until their chances add up to at least $p$, a share you choose, such as 0.9.

Picture packing a suitcase. Top-k says "take exactly five items." Top-p says "take the most useful items until you have 90% of what you need." For a weekend trip, a few items reach 90%. For a long trip, you need many more. In the same way, top-p keeps one or two tokens when the model is sure of itself and many when it isn't. A fixed $k$ can't adapt like that.

Here's top-p on the catalog words, with $p = 0.9$. Add up the chances from the top: light 0.33, then 0.55 with warm, 0.73 with durable, 0.87 with packable. That's still short of 0.9. Adding waterproof brings the total to 0.98, so top-p keeps those five words and drops "purple" and "edible".

import numpy as np

words = ["light", "warm", "durable", "packable", "waterproof", "purple", "edible"]
p = np.exp([2.0, 1.6, 1.4, 1.1, 0.9, -1.0, -2.5])
p /= p.sum()

def top_k(p, k):
    keep = np.zeros_like(p)
    idx = np.argsort(-p)[:k]
    keep[idx] = p[idx]
    return keep / keep.sum()

def top_p(p, threshold):
    order = np.argsort(-p)
    cutoff = np.searchsorted(np.cumsum(p[order]), threshold) + 1   # how many tokens reach the threshold
    keep = np.zeros_like(p)
    keep[order[:cutoff]] = p[order[:cutoff]]
    return keep / keep.sum()

for name, q in [("original", p), ("top-k, k=3", top_k(p, 3)), ("top-p, p=0.9", top_p(p, 0.9))]:
    print(f"{name:<13}" + "  ".join(f"{w} {v:.2f}" for w, v in zip(words, q)))

With $k = 3$, only light, warm and durable survive. Their chances grow to 0.45, 0.30 and 0.25 so they add up to 1 again. With $p = 0.9$, five words survive, each a little larger than before.

Each function follows the same three steps. Sort the tokens with np.argsort(-p), which lists them from most to least likely. Copy the ones you keep into an array of zeros. Then divide by the new total to renormalize.

probs = {"a": 0.5, "b": 0.3, "c": 0.2}
keep = sorted(probs, key=probs.get, reverse=True)[:2]
print(round(sum(probs[k] for k in keep), 1))

On a real model

Now use the bigram model from the last lesson. It predicts each character from the one before it. The code builds its table of counts, then writes 60 characters from the start of a product description. It does this five times, with a different strategy each time.

import numpy as np
import requests

text = requests.get("https://api.gradient.test/v1/corpus").json()["text"]
chars = sorted(set(text))
stoi = {c: i for i, c in enumerate(chars)}
ids = np.array([stoi[c] for c in text])
counts = np.full((len(chars), len(chars)), 0.1)
np.add.at(counts, (ids[:-1], ids[1:]), 1)
logits = np.log(counts / counts.sum(axis=1, keepdims=True))

def generate(T=None, k=None, steps=60, seed=0):
    rng = np.random.default_rng(seed)
    cur, out = stoi["\n"], ""
    for _ in range(steps):
        z = logits[cur]
        if T is None:                                   # greedy
            cur = int(np.argmax(z))
        else:
            p = np.exp((z - z.max()) / T)
            if k is not None:
                p[np.argsort(-p)[k:]] = 0
            cur = rng.choice(len(chars), p=p / p.sum())
        out += chars[cur]
    return repr(out)

print("greedy        ", generate())
print("T = 0.5       ", generate(T=0.5))
print("T = 1.0       ", generate(T=1.0))
print("T = 1.0, k = 5", generate(T=1.0, k=5))
print("T = 2.0       ", generate(T=2.0))

A few lines are worth a closer look:

• The table starts every count at 0.1, so no pair of characters ever has a chance of exactly zero.
• np.log turns each row of chances into scores. At $T = 1$, softmax turns those scores straight back into the same chances.
• When T is left out, the code takes the top character every time. That's greedy decoding.
• The seed=0 fixes the random draws, so you see the same text each time you run it.

Compare the five lines. Greedy decoding falls into a loop almost at once. Low temperature repeats the most common bits of text. High temperature produces junk. A moderate temperature with a top-k cutoff keeps the feel of the catalog without the wildest characters. Large models show the same trade-offs. They play out with words and ideas instead of single letters.

In practice

• Open models and many APIs offer settings called temperature, top_p and top_k. Some add others. Min-p keeps only tokens that are at least some fraction as likely as the top one. A repetition penalty lowers the scores of tokens the model has already used. Common starting points are a temperature of 0.7 to 1.0 and a top-p of 0.9 to 0.95.
• Use a low temperature for tasks with one right answer, such as pulling facts out of text (extraction) or sorting it into groups (classification). Use a higher one for brainstorming and creative copy.
• Even temperature 0 isn't perfectly repeatable on most serving systems. The reason is how the computer chips that run models (GPUs) group many requests together, which can change tiny rounding details.
• As Phase 1 noted, the current Claude API version doesn't accept temperature, top_p or top_k. The provider manages sampling for you. You steer behavior through the prompt and the effort setting instead.

For marketers: when you ask for ten subject-line variants, you're relying on sampling. When you need the same classification every time, design for it. Use structured output, clear instructions and evals (tests that score the model's answers). Don't count on randomness being switched off.