Speculative decoding

It works when sampling too

Real chat models don't always take the likeliest token. They sample: they pick at random, using the model's probabilities as weights. Now "the target agrees" gets fuzzy, because the target has no single pick to compare against.

The 2023 papers solved this with an accept-or-reject rule. It makes speculation give exactly the same mix of tokens as the target alone. Three symbols first:

• $x$ is the token the draft proposed.
• $p(x)$ is the target's probability for that token.
• $q(x)$ is the draft's probability for the same token.

Accept the draft's token with this probability:

Here $\min$ means "the smaller of".

In plain words: if the target likes the token at least as much as the draft does, always keep it. If the draft liked it more than the target does, keep it only some of the time. The fraction $p(x)/q(x)$ says how often.

Here is an example with five words a model might use to describe a sleeping bag:

• The draft proposes "packable". The target gives it 0.20, the draft 0.40. 0.20 ÷ 0.40 = 0.5, so you keep it half the time.
• The draft proposes "warm". The target gives it 0.25, the draft 0.10. 0.25 ÷ 0.10 = 2.5. The smaller of 1 and 2.5 is 1, so you always keep it.

p = {"light": 0.40, "warm": 0.25, "packable": 0.20, "durable": 0.10, "roomy": 0.05}
q = {"light": 0.30, "warm": 0.10, "packable": 0.40, "durable": 0.15, "roomy": 0.05}
for word in ["packable", "warm"]:
    print(word, min(1, p[word] / q[word]))

It prints packable 0.5 and warm 1.

When a guess is rejected

A rejected token needs a replacement. It comes from a leftover list: for each token, the part of the target's probability that the draft didn't cover.

Here $\max$ means "the larger of", so any negative result becomes 0. You then normalize the list: divide each number by the list's total, so the list adds up to 1 again.

In plain words: for each word, take how much more the target wants it than the draft offered. If the draft already offered too much, count zero. Scale the list so it adds up to 1, and pick the replacement from it.

With the five words:

• light: 0.40 − 0.30 = 0.10
• warm: 0.25 − 0.10 = 0.15
• packable: 0.20 − 0.40 is below zero, so 0
• durable: 0.10 − 0.15 is below zero, so 0
• roomy: 0.05 − 0.05 = 0

The total is 0.25. Divide each number by 0.25: light becomes 0.4 and warm becomes 0.6. So a rejected token is replaced by "light" 40% of the time and "warm" 60% of the time. Those are the two words the draft under-rated.

p = {"light": 0.40, "warm": 0.25, "packable": 0.20, "durable": 0.10, "roomy": 0.05}
q = {"light": 0.30, "warm": 0.10, "packable": 0.40, "durable": 0.15, "roomy": 0.05}
leftover = {w: max(0, p[w] - q[w]) for w in p}
total = sum(leftover.values())
print(round(total, 2))
print({w: round(v / total, 2) for w, v in leftover.items()})

It prints 0.25, then {'light': 0.4, 'warm': 0.6, 'packable': 0.0, 'durable': 0.0, 'roomy': 0.0}.

print(min(1, 0.1 / 0.4))

Testing the rule 50,000 times

The code below runs the full rule 50,000 times on the same five words. It uses NumPy arrays instead of dictionaries. The draft is noticeably wrong: it over-rates "packable" and under-rates "warm". Compare the last column with the target's column.

import numpy as np

words = ["light", "warm", "packable", "durable", "roomy"]
p = np.array([0.40, 0.25, 0.20, 0.10, 0.05])      # the big (target) model's next-token probabilities
q = np.array([0.30, 0.10, 0.40, 0.15, 0.05])      # the small draft model's guess

rng = np.random.default_rng(0)
def speculative_token():
    x = rng.choice(5, p=q)                          # the draft proposes a token
    if rng.random() < min(1, p[x] / q[x]):          # accept with probability min(1, p/q)
        return x, True
    residual = np.maximum(p - q, 0)                 # otherwise resample from what the draft under-weighted
    return rng.choice(5, p=residual / residual.sum()), False

samples = [speculative_token() for _ in range(50_000)]
freq = np.bincount([s for s, _ in samples], minlength=5) / len(samples)
print(f"{'token':<10} {'target p':>8} {'draft q':>8} {'speculative':>12}")
for w, pi, qi, f in zip(words, p, q, freq):
    print(f"{w:<10} {pi:>8.2f} {qi:>8.2f} {f:>12.3f}")
print(f"\ndraft accepted {np.mean([a for _, a in samples]):.0%} of the time (theory: sum of min(p, q) = {np.minimum(p, q).sum():.0%})")

The last column closely matches the target's, even though every token started as a draft guess. The bad draft changed nothing about the output.

The draft's token was kept 75% of the time. That number is the overlap between the two lists of probabilities:

The symbol $\sum$ (sigma) means "add up over every token $x$".

In plain words: for each word, take the smaller of its two probabilities. Add those up. The more closely the draft imitates the target, the bigger the overlap.

For the five words: 0.30 + 0.10 + 0.20 + 0.10 + 0.05 = 0.75. In Python, the sum is a loop:

p = [0.40, 0.25, 0.20, 0.10, 0.05]
q = [0.30, 0.10, 0.40, 0.15, 0.05]
print(round(sum(min(a, b) for a, b in zip(p, q)), 2))

It prints 0.75. A bigger overlap means more guesses are kept, so each target pass yields more tokens.

Making it pay

• The draft must be much cheaper and fairly accurate. The draft is usually a small model from the same family as the target. It shares the target's tokenizer, the tool that splits text into tokens, so the guesses line up token for token. It's often distilled from the target: trained to imitate it.
• Some methods skip the separate model. Medusa and EAGLE add small extra prediction heads to the target itself. Prompt lookup drafts by copying stretches of the prompt. That works well when the answer quotes its input: summaries, code edits, and RAG answers built from retrieved documents.
• Speed-ups are typically 2–3× for large models. It depends on how predictable the text is. Code and structured output speed up more than creative writing.
• It costs extra compute. Rejected guesses are wasted work, so speculation saves waiting time, not total work. When a server already runs huge batches, the GPU's arithmetic becomes the limit instead of memory. Then speculation helps less, so providers tune it to their traffic.
• Quality doesn't change. Quantization trades a little quality for speed. Speculative decoding is lossless: by design, the output follows the target's probabilities exactly.

For marketers: this is one reason the same model can feel faster from one provider than another, or faster this month than last, without any change in its answers.