Lesson 88 found the chance of an event. A random variable goes further: it's an uncertain number, like a die's face or tomorrow's rentals, described completely by its distribution, the chance of every value it can take. Once you have the distribution, you can answer any question about the number, and you can draw samples from it: pick values at random, with the likely ones picked more often. That last idea is how a language model writes. At every step it has a distribution over the next word, and generating text means sampling from it, over and over. This lesson builds distributions, samples them by hand and in Python, and writes a few words that way.
What you'll be able to do
Describe an uncertain number as a random variable with a distribution: a table or bars that sum to 1
Find the probability of a range of values, and read a curve's area as probability for continuous values
Sample from a distribution with rng.choice, count the samples into a histogram, and watch it approach the bars
Generate text by sampling next words, compare it with always taking the most likely one, and reshape the odds with temperature and top-k in PyTorch
Quick check
Have a guess before you start. A spinner lands on 1 half the time, 2 a third of the time and 3 otherwise. What is P(3)?
1/2
1
1/6
1/3
Show the answer
1/6
The chances must add to 1: 1 − 1/2 − 1/3 = 1/6.
The math: an uncertain number and its distribution
Purpose. A full description of an uncertain number.
A random variable X is a number whose value depends on chance: the face of a die, the number of heads in ten tosses, the next word's position in a vocabulary. Its distribution gives the probability of every value:
x
1
2
3
4
P(X = x)
0.5
0.3
0.15
0.05
In plain words: a table (or a bar chart) of each value and its chance. The chances are between 0 and 1 and add up to 1.
Questions about X become sums: P(X ≥ 3) = 0.15 + 0.05 = 0.2; P(X is even) = 0.3 + 0.05 = 0.35.
Quick check
From lesson 88: in that table, what is P(X ≠ 1)?
0.5
0.3
1
0
Show the answer
0.5
The complement: 1 − P(X = 1) = 1 − 0.5 = 0.5.
Sampling. To draw a value at random from the distribution, give each value a slice of the numbers from 0 to 1, as wide as its probability. Then pick a uniform random number u, one that is equally likely to land anywhere from 0 up to 1, and see whose slice it lands in. Running totals mark the slices' edges.
The picture. Step through it with u = 0.65. The picture draws the numbers from 0 to 1 as a line, with u as a ring on it. Each step jumps along the line by the next word's chance, from one running total to the next, so each jump spans that word's slice. Watch for the jump that carries the walk past u: its word is the sample.
Try it now
Then set the random number u to 0.1 and press Use these numbers: the ring lands in fire's slice. Try 0.9 too: tent's. Across many random u, each word is picked as often as its probability.
Symbols and formula. A distribution's probabilities sum to 1, and a range's probability sums its values':
x∑P(X=x)=1P(a≤X≤b)=a≤x≤b∑P(X=x)
In plain words: add the bars in the range.
Continuous values. Some quantities, like a temperature, can take any value in a range. Then no single value has a chance of its own; instead a density curve is drawn, and probability is the area under it over a range, lesson 84's integral. The total area is 1, just as the bars add to 1.
Quick check
In the table above, what is P(2 ≤ X ≤ 3)?
0.3
0.45
0.5
0.15
Show the answer
0.45
Add the bars for 2 and 3: 0.3 + 0.15 = 0.45.
In your own words
In your own words: why does slicing [0, 1) by running totals make each value come up as often as its probability?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: rng.choice, and a histogram
Purpose. Sample from a distribution in one call, and check it with counts.
import numpy as np
rng = np.random.default_rng(0)
p = np.array([0.5, 0.3, 0.15, 0.05])
for n in [20, 2000]:
s = rng.choice(4, size=n, p=p)
print(n, (np.bincount(s, minlength=4) / n).round(3).tolist())
It runs right here in your browser, with nothing to install.
rng.choice(4, size=n, p=p) draws n values from 0, 1, 2, 3 with the given probabilities. np.bincount counts how many times each value came up, and dividing by n turns counts into shares: a histogram. With 20 samples the shares are rough (value 3 might not appear at all); with 2,000 they're close to the bars.
Code flow.np.bincount does its counting out of sight. Here is the same counting as a loop you can step through: six samples with seed 0 (the first six of the twenty above), and a list counts with one tally box for each value, 0 to 3. Watch each sample add 1 to its own box: a histogram filling as samples arrive.
A named mistake: probabilities that don't add to 1.rng.choice refuses a p that doesn't sum to 1, with "probabilities do not sum to 1". Counts must be turned into shares first (lesson 88), and rounded probabilities can drift: 0.33 + 0.33 + 0.33 is 0.99. Divide by the sum to fix both.
Sampling words directly.rng.choice also accepts a list of words, returning words instead of positions, which is how the next part writes text.
Worked example
Worked example: a weighted die, 20 and 2,000 times
A die is weighted so that 6 comes up half the time and every other face 0.1. Sample it 20 and 2,000 times and compare the shares with the table.
import numpy as np
rng = np.random.default_rng(4)
p = np.array([0.1, 0.1, 0.1, 0.1, 0.1, 0.5])
faces = np.array([1, 2, 3, 4, 5, 6])
for n in [20, 2000]:
s = rng.choice(faces, size=n, p=p)
print(n, round(float(np.mean(s == 6)), 3), round(float(np.mean(s == 1)), 3))
print(p.sum())
In the lesson you can change this and run it.
With 20 rolls the share of sixes can easily land 0.15 away from 0.5; with 2,000 it's within about 0.02. The last line checks the table: its probabilities add to 1, as rng.choice requires. Small samples are noisy; large ones settle on the distribution, the law of large numbers (lesson 98).
Quick check
What does this print?
import numpy as np
s = np.array([0, 2, 1, 0, 0, 2])
print(np.bincount(s, minlength=3).tolist(), (np.bincount(s, minlength=3) / 6).round(2).tolist())
[3, 1, 2] [3, 1, 2]
[2, 1, 3] [0.33, 0.17, 0.5]
[3, 1, 2] [0.5, 0.17, 0.33]
[0, 2, 1] [0.0, 0.33, 0.17]
Show the answer
[3, 1, 2] [0.5, 0.17, 0.33]
Value 0 appears 3 times, 1 once and 2 twice. Divided by 6: 0.5, 0.17 and 0.33.
Drill
Counts into probabilities, then sample
The counts below don't add to 1, so rng.choice would refuse them. Fix probs so it does, then sample 5,000 values with seed 2 and set share2 to the share of 2s, rounded to 2 places.
In the lesson, your code is checked as you write it.
In your own words
Why do 20 samples rarely match the distribution's bars, while 2,000 usually do?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: generating text is sampling
Purpose. A language model writes by sampling, one word at a time.
At each step a model gives a distribution over the next word. Sampling picks a word with those odds; taking the argmax always picks the single most likely one:
wnext∼P(w∣context)orwnext=argwmaxP(w∣context)
In plain words: "∼" means "drawn from". argmax means "the one with the biggest probability".
import numpy as np
words = ["fire", "tent", "trail"]
p = [0.6, 0.2, 0.2]
rng = np.random.default_rng(3)
print(rng.choice(words, size=10, p=p).tolist())
print(words[int(np.argmax(p))])
Ten samples are mostly "fire" with some variety; argmax gives "fire" every time. That's why sampled text varies and argmax text repeats itself.
Quick check
A model always takes the argmax next word. What tends to happen to its writing?
It becomes random, like rolling dice
It stops after one word
It uses rare words more often than it should
It's predictable and can loop on the same phrases
Show the answer
It's predictable and can loop on the same phrases
Always taking the most likely word removes all variety, so the same context gives the same words, often repeating.
Slices, in code.rng.choice does with numbers what the flow did on the line. The running totals of the chances, which np.cumsum computes, are the slice boundaries: for chances 0.6, 0.3 and 0.1 they're 0.6, 0.9 and 1.0, so a uniform u below 0.6 picks the first word, a u from 0.6 up to 0.9 the second, and the rest the third. Each word wins as often as its slice is wide. np.argmax ignores the slices entirely: it returns the position of the biggest chance, the same answer every time, however close the runner-up. Predict both printouts first.
Quick check
What does this print?
import numpy as np
p = np.array([0.6, 0.3, 0.1])
print(int(np.argmax(p)), np.cumsum(p).round(2).tolist())
2 [0.6, 0.9, 1.0]
0 [0.6, 0.9, 1.0]
0.6 [0.6, 0.9, 1.0]
0 [0.6, 0.3, 0.1]
Show the answer
0 [0.6, 0.9, 1.0]
argmax gives the position of the biggest chance, 0; the running totals 0.6, 0.9 and 1.0 are the slice boundaries.
In PyTorch. The same calls on a tensor. torch.argmax and .cumsum(0) do what NumPy's argmax and cumsum did. torch.multinomial(p, n) draws n positions, each as often as its chance: PyTorch's rng.choice. replacement=True puts each drawn position back, so it can come up again.
import torch
p = torch.tensor([0.6, 0.3, 0.1])
print(torch.argmax(p).item(), p.cumsum(0))
draws = torch.multinomial(p, 10000, replacement=True)
print((torch.bincount(draws, minlength=3) / 10000).round(decimals=1))
The first line prints 0 and the slice edges 0.6, 0.9 and 1.0: the numbers the NumPy check above printed, now as a tensor. The second shares out 10,000 draws, rounded to one place: 0.6, 0.3 and 0.1, the chances themselves. A language model calls torch.multinomial(probs, 1) once for every word it writes.
Temperature and top-k. Real systems reshape the distribution before sampling. A model's last layer gives each word a score, and softmax turns scores into chances (lesson 104 builds it): it takes e to the power of each score, then shares the results out so they add to 1. The temperature T divides the scores first. Here the scores are the logs of the chances, and e to the log of a chance is the chance, so at T = 1 softmax gives the chances back:
import torch
import torch.nn.functional as F
p = torch.tensor([0.6, 0.3, 0.1])
scores = torch.log(p)
for T in [0.5, 1.0, 2.0]:
print(T, F.softmax(scores / T, dim=0).round(decimals=2))
top = p.topk(2)
print(top.values, top.indices)
A low temperature sharpens the chances toward the argmax: at 0.5, the first word's 0.6 grows to 0.78. A high one flattens them toward even: at 2, it shrinks to 0.47. Top-k keeps only the k most likely words and shares their chances out again so they add to 1: p.topk(2) finds the two biggest chances, 0.6 and 0.3, at positions 0 and 1. Both change the widths of the slices before the draw; the draw itself stays the same. The sampler widget in the math part has both controls.
Quick check
At temperature 0.5, the chances 0.6, 0.3 and 0.1 became 0.78, 0.2 and 0.02. What happens to text sampled from them?
Nothing changes: temperature acts on scores
It varies more, closer to rolling a fair die
The first word comes up more often, the third almost never
It's the argmax word every time, never another
Show the answer
The first word comes up more often, the third almost never
Sharper chances mean the likely word wins more often: 78 draws in 100 instead of 60, and the third word only about 2. The text gets safer and more repetitive, still with a little variety.
Drill
Sample a few words
Using the next-word distribution below, set sampled to 8 words sampled with seed 11 (as a list), top to the argmax word, and share_top to the share of sampled that equals top.
import numpy as np
words = ["tent", "stove", "map", "rope"]
p = [0.4, 0.3, 0.2, 0.1]
sampled = []
top = ""
share_top = 0print(sampled, top, share_top)
In the lesson, your code is checked as you write it.
In your own words
Why would you sample instead of always taking the most likely word?
Write a sentence or two, then compare it with a good answer in the lesson.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write