Every chat API has a temperature setting, and most people turn it by folklore: "lower for facts, higher for creativity". The sampling lesson gave the formula. This lesson shows the behavior behind the folklore on the course's tiny GPT: at 0 the same prompt gives the same words every time and sometimes gets stuck in a loop; at 1.5 the text starts breaking grammar; by 2 more than a third of the adjacent word pairs have never appeared in the training text.
Those are measurable, and you will measure them with two numbers you can compute on any model's output: the share of adjacent token pairs the model never saw in training, and the share of repeated pairs. With them, choosing a temperature stops being folklore and becomes a setting you can justify.
What you'll be able to do
Show what the same prompt produces at temperature 0, 0.7 and 1.5 on a real model
Explain temperature as a division of the logits before softmax, and read its effect on entropy
Measure nonsense and repetition with two simple rates, and see how they move with temperature
Choose a temperature for a task, and know when top-p or a repetition penalty is the better tool
Quick check
The same prompt is sent to a model at temperature 0 and at temperature 2. A guess at what differs in the output:
Temperature changes which words the model knows, adding rare ones at high values
At 2 the model is more accurate because it considers more options before choosing
At 0 the model refuses to answer, since no randomness means no output
At 2 the text often breaks grammar and invents word pairs; at 0 it repeats itself
Show the answer
At 2 the text often breaks grammar and invents word pairs; at 0 it repeats itself
Temperature reshapes the probabilities before a token is drawn. High values flatten them, so unlikely, ungrammatical continuations get drawn; zero always takes the top token, which is deterministic and can circle back on itself.
The idea: temperature divides the logits
Purpose. What the temperature knob does to a distribution, and how to read the change in entropy.
Open any chat API and there it is: temperature, from 0 to about 2. Sampling strategies gave the definition. The model produces a logit for every token. Dividing all the logits by the temperature T before the softmax sharpens the distribution when T<1 and flattens it when T>1. At T=0 the top token is taken outright.
In plain words: divide every score by T before the softmax. A small T widens the gaps between scores, so one token wins more often. A large T narrows them, so the choice spreads out.
What the definition does not show is behavior: what text actually comes out at each setting, and how you would tell a good setting from a bad one without reading a thousand samples. That is what this lesson measures, on the course's tiny GPT. Its small size is an advantage here: you can generate hundreds of tokens in a second and compare them with every pair of words it was trained on.
Worked example
One prompt, three temperatures
Thirty tokens after "Our Kestrel tent is", with the random seed fixed so the draws repeat.
import torch
from tinygpt import load_pretrained
model, tok = load_pretrained("tiny-gpt")
prompt = torch.tensor([tok.encode("Our Kestrel tent is")])
for T in [0, 0.7, 1.5]:
torch.manual_seed(0)
out = model.generate(prompt, 30, temperature=T)[0, prompt.shape[1]:].tolist()
print(f"T={T}: {tok.decode(out)!r}")
It runs right here in your browser, with nothing to install.
At 0 the model writes a correct catalog entry and moves on to the next product, the most likely path at every step. At 0.7 it writes a different, equally plausible entry. At 1.5 the grammar gives way within a few words: "made from for Canyon trails", "bright on the promise. trade travel". Nothing about the model changed between the three runs; the shares each token got before the draw did.
Quick check
The tiny GPT's top three logits after "Our Kestrel tent is made from" are 12.0 (silnylon), 11.88 (recycled) and 7.18 (waxed). What does this print?
At T = 1 silnylon has 0.53. Dividing by 2 halves every gap between logits, so the two close candidates move toward equal shares and silnylon drops to 0.49, while waxed gains a little.
Sharpen or flatten
Dividing the logits by T multiplies every gap between them by 1/T. The gap between silnylon and recycled is 0.12 logits, which softmax turns into shares of 0.53 and 0.46 at T=1. At T=0.5 the gap doubles and silnylon pulls ahead; at T=2 the gap halves and the two almost tie, while the long tail of tokens with logits around 5 to 7 gains share it did not have.
The single number that captures this is the entropy of the distribution (Surprise, entropy and cross-entropy): low when one token dominates, high when many share. The next cell computes it for the "made from" distribution at three temperatures. Watch it climb from about one bit, a near coin flip between two materials, to three and a half bits, a spread over many tokens that includes the ones that break the sentence.
Worked example
Entropy at three temperatures
The full distribution over all 195 tokens after "Our Kestrel tent is made from", at T = 0.5, 1 and 2: top share and entropy in bits.
import math
import torch
from tinygpt import load_pretrained
model, tok = load_pretrained("tiny-gpt")
with torch.no_grad():
logits = model(torch.tensor([tok.encode("Our Kestrel tent is made from")]))[0, -1]
for T in [0.5, 1.0, 2.0]:
p = torch.softmax(logits / T, dim=-1)
nonzero = p[p > 0]
entropy = -float((nonzero * nonzero.log()).sum()) / math.log(2)
print(f"T={T}: top share {p.max().item():.2f}, entropy {entropy:.2f} bits")
In the lesson you can change this and run it.
At 0.5 the top share is 0.56 and the entropy 0.99 bits: effectively one binary choice between the two tent materials. At 1 it is 0.53 and 1.12 bits. At 2 the top share falls to 0.35 and the entropy jumps to 3.55 bits, which is the spread of about eleven equally likely tokens. The extra bits are the tail: "for", "light", "pitch" and dozens of other words that do not belong after "made from".
Try it now
The tiny GPT's own next-token shares after "It is light and", a wide distribution. Move the temperature and watch the bars; press Sample to draw.
In your own words
In your own words: why does dividing the logits by a number above 1 make ungrammatical continuations more likely?
Write a sentence or two, then compare it with a good answer in the lesson.
In code: measure nonsense and loops
Purpose. Turn "the text is breaking" into two rates you can compute from the tokens alone.
Reading samples does not scale. You need a number that rises when text breaks, computed from the tokens alone. A simple one works well on this model: the share of adjacent token pairs that never appear in the training text. Every description the tiny GPT learned from is available, so the set of pairs it has seen is exact. A pair like ("made", "from") is in it; ("from", "for") is not. Fluent output mostly reuses seen pairs; nonsense invents new ones.
A second number catches the opposite failure, repetition: the share of adjacent pairs that already appeared earlier in the same sample. The next cell draws eight samples of thirty tokens at each of five temperatures and reports both rates. Eight short samples are enough to see the trend; in practice you would use more.
Code flow. The unseen-pair rate is a set lookup. Step through it on a four-word output and two pairs the model saw in training.
Worked example
Unseen pairs and repeated pairs, by temperature
The pairs the model saw in training come from the corpus endpoint. Eight samples of thirty tokens per temperature (one at T = 0, which is deterministic), seeds 0 to 7.
import re
import torch
import requests
from tinygpt import load_pretrained
model, tok = load_pretrained("tiny-gpt")
text = requests.get("https://api.gradient.test/v1/corpus").json()["text"]
seen = set()
for line in text.strip().split("\n"):
words = tok.tokenize(line)
seen.update(zip(words, words[1:]))
print(f"{len(seen)} different adjacent pairs in the training text")
prompt = torch.tensor([tok.encode("Our Kestrel tent is")])
temps = [0, 0.5, 1.0, 1.5, 2.0]
unseen_rates, repeat_rates = [], []
for T in temps:
unseen = repeated = pairs = 0for seed inrange(1if T == 0else8):
torch.manual_seed(seed)
out = [tok.vocab[i]
for i in model.generate(prompt, 30,
temperature=T)[0, prompt.shape[1]:].tolist()]
bigrams = list(zip(out, out[1:]))
pairs += len(bigrams)
unseen += sum(1for b in bigrams if b notin seen)
repeated += len(bigrams) - len(set(bigrams))
print(f"T={T}: unseen pairs {unseen / pairs:.0%}, repeated pairs "f"{repeated / pairs:.0%}")
unseen_rates.append(unseen / pairs)
repeat_rates.append(repeated / pairs)
import matplotlib.pyplot as plt
plt.plot(temps, unseen_rates, "o-", label="unseen pairs")
plt.plot(temps, repeat_rates, "s--", label="repeated pairs")
plt.xlabel("temperature")
plt.ylabel("share of adjacent pairs")
plt.title("Nonsense rises with temperature")
plt.legend()
plt.show()
In the lesson you can change this and run it.
The training text has 634 different adjacent pairs. From 0 to 1 the model stays inside them: 7%, 6% and 4% of its pairs are new, and the few that are new tend to be sentence boundaries the catalog never had. At 1.5 the rate jumps to 16%, and at 2 to 38%: more than a third of what it writes was never seen as a pair. Repetition peaks at 1.5 (12%), where the model stumbles into loops of a few words, and is low at 2 because the text is too scattered to repeat anything.
Quick check
At temperature 2 the tiny GPT's unseen-pair rate is 38%, against 4% at temperature 1. What is the best reading?
The 4% at temperature 1 shows the model copying the training text word for word
At 2 the model has learned new vocabulary from the prompt and uses it in new ways
Unseen pairs are a sign of creativity, and the higher the rate the better the text
At 2 the flattened distribution hands real probability to tokens that never follow their predecessors
Show the answer
At 2 the flattened distribution hands real probability to tokens that never follow their predecessors
Unseen pairs are mostly ungrammatical joins like "from for". Their rate tracks how much share the tail of the distribution gets, which is what temperature controls.
Quick check
From the sampling lesson: what does top-p do that temperature does not?
It cuts the tail: only the smallest set of tokens whose probabilities reach p can be drawn
It changes the logits before the softmax, the way temperature does
It raises the probability of the top token by a fixed amount set by p
It makes decoding deterministic, so the same prompt always gives the same text
Show the answer
It cuts the tail: only the smallest set of tokens whose probabilities reach p can be drawn
Temperature reshapes the whole distribution; top-p removes the tail entirely and renormalizes what is left. The two are often combined: a temperature for variety, top-p to keep the nonsense out.
Drill
Share of unseen pairs
Write unseen_rate(tokens, seen): the share of adjacent pairs in tokens that are not in the set seen. Return 0.0 when there are fewer than two tokens.
In the lesson, your code is checked as you write it.
Loops at zero
Temperature 0 has its own failure. Greedy decoding takes the top token at every step and feeds it back in, so the process is deterministic: the same context always produces the same next token. If the model ever returns to a context it has seen a few tokens earlier, it will produce the same continuation again, and the text circles. The tiny GPT does it within sixty tokens: "Made for cold for cold nights" is the start of one, and so are the "Meet the Harbor" openings that keep coming back.
Large models loop too, in longer cycles: a sentence that restates the previous one, a list that repeats its items. This is why decoding settings include a repetition penalty or a frequency penalty, which lowers the logits of tokens that have already appeared, and why a little randomness (a temperature around 0.7) breaks the cycle on its own. The next cell shows the loop and counts its repeated pairs.
Worked example
Sixty greedy tokens
Temperature 0, sixty tokens, and the share of adjacent pairs that already occurred earlier in the same output.
import torch
from tinygpt import load_pretrained
model, tok = load_pretrained("tiny-gpt")
prompt = torch.tensor([tok.encode("Our Kestrel tent is")])
out = [tok.vocab[i]
for i in model.generate(prompt, 60, temperature=0)[0, prompt.shape[1]:].tolist()]
bigrams = list(zip(out, out[1:]))
print(tok.decode([tok.token_id(w) for w in out]))
print(f"repeated pairs: {len(bigrams) - len(set(bigrams))} of {len(bigrams)}, "f"{(len(bigrams) - len(set(bigrams))) / len(bigrams):.0%}")
In the lesson you can change this and run it.
The first sentences are fine. Then "Made for cold for cold nights", a two-word cycle the greedy path fell into and left, and later "Meet the packable. Meet the Harbor", the model restarting the catalog's opening because that is the most likely thing after a full stop. Eleven of the 59 pairs, 19%, are repeats. Greedy decoding has no memory of what it already wrote beyond the context itself, and the context keeps steering it back to the same high-probability openings.
Quick check
Why can greedy decoding (temperature 0) produce a loop like "cold for cold for cold"?
The model is trained to repeat itself at temperature 0 as a sign of confidence
Greedy decoding ignores the prompt and produces the most common tokens in order
It is deterministic: the same context gives the same next token, so a repeated context repeats
The random number generator returns the same number every time at temperature 0
Show the answer
It is deterministic: the same context gives the same next token, so a repeated context repeats
With no randomness, the model's output is a function of its context. If the last few tokens recreate an earlier context, the function produces the same continuation, and the cycle closes.
Drill
Share of repeated pairs
Write repeat_rate(tokens): the share of adjacent pairs that already occurred earlier in the same list. Return 0.0 when there are fewer than two tokens.
In the lesson, your code is checked as you write it.
In your own words
A colleague wants to use temperature 0 for a product-description generator "so it never says anything strange". What two problems would you warn about?
Write a sentence or two, then compare it with a good answer in the lesson.
In production: what this justifies
Purpose. Pick a temperature for a task, and know the decoding settings that sit beside it.
Pick the temperature by the task, then measure. Extraction, classification, code and anything with one right answer: 0 to 0.3, where the unseen-pair rate was lowest here, and where you can reproduce results. Drafting and ideas: 0.7 to 1, which still stayed inside the model's known pairs. Above about 1.3, expect broken text; on the tiny GPT 1.5 quadrupled the nonsense rate. Production models are more robust, but the curve has the same shape, and your evaluation set should be run at the temperature you deploy.
Pair temperature with top-p. A high temperature for variety and top-p to remove the tail gives the variety without the broken joins, because the tokens that make unseen pairs live in the tail.
Treat repetition as a decoding problem. A loop is not the model "not knowing more"; it is deterministic decoding meeting a repeated context. Randomness or a repetition penalty fixes it; a bigger model may not.
Quick check
A deployment uses temperature 1.8 "for creativity". What does this print, and what does it suggest about the setting?
From temperature 1 to 2 the unseen-pair rate grew about tenfold on the tiny GPT (0.04 to 0.38), and 1.8 sits well past the 1.5 where the breakage began. Variety that keeps grammar comes from a moderate temperature with top-p, not from 1.8.
Where this shows up in an engineer's week
A product writer says the descriptions have gone strange, or that they repeat. Check the deployed temperature first, then run your eval set at that setting, since a score measured at 0 says little about 1.5. A low temperature with a repetition penalty fixes loops. A high temperature with top-p keeps variety without broken joins.
11DistillationTrain a small model to imitate a large one.35 min
12Tuning an embedding model for retrievalWhen search fails on your own vocabulary: train an adapter with a contrastive loss on question and passage pairs, test on held-out passages, use batch size and hard negatives, and measure what tuning costs elsewhere.45 min
13QuantizationStore weights in fewer bits to run models cheaply.35 min
23Copying from the promptIn-context learning as a two-head copying circuit: the base tiny GPT cannot copy a repeated pattern, a model trained on repeats grows an induction head and can.30 min
24Why temperature changes the answerThe same prompt at 0, 0.7 and 1.5: deterministic loops, plausible variety, broken grammar, measured on a real model.25 min
25Why long prompts diluteAttention is a budget shared by every token, so a fact far from the question loses its share; measured, with the cliff beyond the trained window.25 min
26Why it never says "I don't know"A chat model answers every request because its fine-tuning data always did; preference training adds flattery; how to build abstention in.25 min
32Weight-space programming: what it is, and what has been shownFreeOptionalA language model reads and writes another network's weights with no training loop: the published experiments, their limits and the criticism, and one edit you make by hand.30 min
33Anatomy of a checkpointOptionalA model file is a set of named tensors: count them, find the ones that write to the residual stream, and save one to bytes and back.35 min
34Reading weights without running the modelOptionalSingular values, and an attention head's token-to-token circuit, read from the matrices alone and then checked against a measurement.35 min
35Writing weights by handOptionalSet a small network's 55 weights so it catches every ball in a paddle game, the way a program would be written, and meet compiled transformers.40 min
36Edits are deltas: task vectors, patches and hashesOptionalSubtract a base from a fine-tune and the edit becomes an object: scale it, take it away, ship it as a patch, and prove by hash what was applied.35 min
37Merging modelsOptionalAverage two fine-tunes, add their task vectors, try SLERP, TIES and DARE, measure the interference, and compare with a model trained on both tasks.40 min
38Low-rank editsOptionalCut a fine-tune's change down to its largest directions with the SVD: how few a behavior needs, and how to pull an adapter out of a delta.35 min
39Directions: steer the run or edit the weightsOptionalFind a behavior's direction in the residual stream, add it at run time, project it out of the weights, and see the effect change from input to input.40 min
40Editing one factOptionalA rank-one change to one matrix rewrites what a product is made from; then reliability, generalization, locality, and what breaks after many edits.40 min
41The editor's harnessOptionalWhat an LLM that edits weights is given and what it is denied: weights as text and their cost in tokens, a sandbox, a budget of forward passes, and rules the harness enforces.40 min
42An LLM reads a network's weightsOptionalSix recorded sessions: an LLM is handed 1,315 weights as text and asked what the network was trained to do. Score each answer against the network, and study one it got wrong.40 min
43An LLM fixes a model's weightsOptionalFind a tiny GPT's loop by hand, then replay recorded sessions in which an LLM hunts for it with ten forward passes, and measure every fix on held-out questions.45 min
44An LLM writes a network's weightsOptionalFive recorded sessions: an LLM writes every weight of a paddle network from the game's source and one sentence. Score each against its specification, test its design claims, and edit one by hand.40 min
45Did it work? Testing an editOptionalReview a patch for a tiny GPT that can't stop talking: held-out questions, a paired comparison, a same-size control, and what the edit cost.40 min
46The published edit, on a real modelOptionalRead the edit published for an open model of 870 million weights, apply and verify one in the same form, see what 16-bit and 4-bit storage do to a fix, and run the real thing from a notebook.45 min
47Safety, licenses and provenanceOptionalBefore a custom model is released: trace its lineage by hash, work out what its parents' licenses pass on, see what an edit can do to a safeguard, and write its card.40 min
48Capstone: ship a custom modelOptionalSet release gates, then diagnose, edit, measure and document a tiny GPT with a fault no lesson has fixed, and write the record a stranger could check.45 min
Confident wrong answers, miscounted letters, copying from the prompt, temperature, long prompts that dilute, and a model that never says "I don't know": each behavior reproduced on a tiny GPT with every number printed by code, then the mechanism behind it and the Phase 1 technique it justifies.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write