A probability is a number for how likely something is, from 0 (impossible) to 1 (certain). For simple experiments, like rolling a die, it comes straight from counting: the ways the thing can happen, divided by all the ways anything can happen. When counting is hard, you can run the experiment many times and see how often it happens. Both ideas sit at the center of machine learning: a language model's entire output is a list of probabilities, one for each possible next word, that add up to 1. This lesson builds probability from counts, checks it by simulation, and turns a camp log into the kind of next-word table a model produces.
What you'll be able to do
Find a probability as a share: the ways an event can happen over all the equally likely ways
Use the rules every probability obeys: between 0 and 1, all outcomes adding to 1, and P(not A) = 1 − P(A)
Estimate a chance by simulation with a seeded random generator, the Monte Carlo method, and watch more trials settle
Turn word counts into a next-word distribution, the whole output of a language model, in Python and with PyTorch's counts / counts.sum()
Quick check
Have a guess before you start. A fair die is rolled. What is the chance of an even number?
1/2
2/3
1/6
1/3
Show the answer
1/2
3 of the 6 equally likely faces are even (2, 4, 6): 3/6 = 1/2.
The math: a share of the possibilities
Purpose. A number for how likely.
An experiment has outcomes, like the six faces of a die. An event is a set of outcomes, like "even". When the outcomes are equally likely:
P(A)=all waysways A can happen
In plain words: the share of the possibilities that count as A.
P(even) = 3/6 = 0.5. A bag of 10 cards with 4 red: P(red) = 4/10 = 0.4. Every probability is between 0 and 1, and the probabilities of all the outcomes add up to 1: something has to happen.
Quick check
From lesson 87: how many outcomes are there when two dice are rolled?
36
12
11
6
Show the answer
36
6 faces for the first die times 6 for the second: 36 equally likely pairs.
From counts to a distribution. A table of counts becomes a table of probabilities by dividing each count by the total. Here a camp log was searched for the word after "the": fire 3 times, tent once, trail once.
The picture. Each count is a bar, and a fourth bar, total, will hold all of them put together. Dividing a count by the total asks how far up the total bar its own bar reaches. Step through and watch each bar trade its count for its share. No bar changes height: only the numbers on them change.
Try it now
Then change the counts to 6, 3, 1 and press Use these numbers: the shares change, but they still add to 1.
Symbols and formula. Two rules that hold for every probability:
0≤P(A)≤1P(not A)=1−P(A)
In plain words: a chance is never below 0 or above 1, and the chance something doesn't happen is 1 minus the chance it does.
The complement trick. "At least one" questions are often easier through "none". The chance of at least one six in two rolls is 1 minus the chance of no six: 1 − (5/6) × (5/6) = 11/36 ≈ 0.306. Listing every way to get at least one six works too, but it's slower and easier to get wrong.
Quick check
A bag has 10 cards, 4 red and 6 black. What is the chance a card drawn is not red?
0.5
0.4
6
0.6
Show the answer
0.6
P(not red) = 1 − 0.4 = 0.6, the same as 6 black out of 10.
Quick check
Using the complement, what is the chance of at least one six in two rolls of a die?
1/36
2/6
25/36
11/36
Show the answer
11/36
No six twice: (5/6)² = 25/36. So at least one six: 1 − 25/36 = 11/36.
In your own words
In your own words: why does "at least one" usually go faster through the complement?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: simulate, then compare with the theory
Purpose. Measure a chance by repeating an experiment, and see the measurement settle.
import numpy as np
rng = np.random.default_rng(0)
for n in [60, 600, 60000]:
rolls = rng.integers(1, 7, size=n)
print(n, round(float(np.mean(rolls % 2 == 0)), 3))
It runs right here in your browser, with nothing to install.
np.random.default_rng(0) makes a random generator with a seed, 0, so the "random" numbers come out the same every run. rng.integers(1, 7, size=n) rolls n dice (the 7 isn't included). rolls % 2 == 0 is True for even rolls, and np.mean of True and False values is the share of Trues. The share drifts toward 0.5 as n grows.
Code flow. Here is a run small enough to watch: 6 rolls from the same seed, 0, and a loop that counts the even ones. rolls shows as a row of six boxes. Step through and watch evens go up each time a roll is even.
A named mistake: no seed. Without a seed, every run gives different numbers, so a result can't be repeated or checked, and a bug can vanish when you look for it. Seed the generator once at the top, and the whole run is reproducible. Different seeds give different but equally valid samples.
How close is close? A simulation's share wobbles around the true chance, and the wobble shrinks as the number of trials grows: roughly ten times the trials for about a third of the wobble (lesson 98 says exactly why). In the cell above, 60 rolls gave 0.517, off by 0.017; 60,000 rolls gave 0.499, within a thousandth.
Watching it settle. The plot follows one run of 1,000 rolls from the same seed. np.cumsum keeps a running count of the even rolls (lesson 84), and n = np.arange(1, 1001) counts the rolls so far, 1 to 1,000, so dividing one by the other gives the share after every roll, drawn against n:
import numpy as np
import matplotlib.pyplot as plt
rng = np.random.default_rng(0)
rolls = rng.integers(1, 7, size=1000)
n = np.arange(1, 1001)
plt.plot(n, np.cumsum(rolls % 2 == 0) / n)
plt.xlabel("rolls so far")
plt.ylabel("share of even rolls")
plt.show()
Its first six rolls are the six from the code flow. Estimating a chance this way, by running a random experiment many times and counting, is called the Monte Carlo method, after the casino district of Monaco. It's the way in whenever counting the possibilities is too hard.
Quick check
Read the plot. Where does the share start, and where does it end up?
At 0.5, and it stays flat the whole way
At 0, then it climbs steadily to 1
At 1, then it settles near 0.5
At 1, and it stays near 1
Show the answer
At 1, then it settles near 0.5
The first five rolls are all even, so the share starts at 1. The more rolls so far, the less one new roll can move it, and it settles near 0.5, the true chance of an even roll: big swings early, small wobbles late.
Worked example
Worked example: two events, simulated and computed
Simulate 100,000 pairs of dice. Estimate the chance of at least one six and of a total of 7, and compare with the exact answers.
| means "or" element by element: True where either die shows a six. Both simulated shares land within about 0.003 of the exact answers, 0.306 and 0.167. Simulation is how you check a probability you're unsure of, and how you estimate ones too messy to count.
4 of the 6 rolls are even (2, 6, 4, 6) and 2 are sixes: shares 4/6 and 2/6.
Drill
Simulate a chance
Seed the generator with 3, roll 50,000 dice, and set est to the share of rolls that are 5 or 6, rounded to 2 places. Then set exact to the true chance, rounded to 2 places, and close to whether they're within 0.01.
import numpy as np
rng = np.random.default_rng()
rolls = rng.integers(1, 7, size=50000)
est = 0
exact = 0
close = Falseprint(est, exact, close)
In the lesson, your code is checked as you write it.
In your own words
Why seed a random generator when you could just run the simulation again?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: a next-word distribution
Purpose. A language model's whole output is a probability distribution.
Count which word follows "the" in a short camp log, then divide by the total. The result is a distribution: one probability per word, adding to 1. A model predicts exactly this kind of table for every next word:
P(w∣"the")=count("the")count("the"w)
In plain words: of all the times "the" appeared, what share was followed by each word.
log = "the fire was warm the tent was dry the fire died the trail was long the fire came back".split()
names = []
counts = []
for i inrange(len(log) - 1):
if log[i] == "the":
w = log[i + 1]
if w in names:
counts[names.index(w)] = counts[names.index(w)] + 1else:
names.append(w)
counts.append(1)
total = sum(counts)
for i inrange(len(names)):
print(names[i], counts[i] / total)
.split() cuts the text into words. Two lists side by side hold each next word and its count; names.index(w) finds where w is. After "the": fire 0.6, tent 0.2, trail 0.2.
Quick check
In the same log, every word after "was" (warm, dry, long) appears once. What is P(warm | "was")?
1
1/5
1/3
1/16
Show the answer
1/3
"was" is followed by three different words, once each: each gets 1/3.
In PyTorch. A model's output is a tensor with one number per word. Put the counts after "the" in a tensor, with a fourth word the club might write, stove, which never follows "the" in the log: a count of 0. One line, counts / counts.sum(), divides every count by the total at once, as the flow's PyTorch tab does:
The same 0.6, 0.2 and 0.2 the list cell printed, now with stove's 0, and probs.sum() confirms they add to 1 (PyTorch writes 1.0 as 1.). The picture draws probs as bars.
From counting to modeling. Counting pairs like this is a bigram model, the simplest language model. It can only predict words it has seen after a given word, and it never generalizes: a log without "the stove" gives "stove" a probability of 0 after "the". Neural language models learn to give sensible probabilities even to combinations they've never seen, but their output is still the same kind of object: a list of chances that add to 1.
Reading a model's distribution. Each row of a next-word table is a distribution, so the rules from the first part apply to it directly: every chance lies between 0 and 1, and the chances add up to 1. The complement rule reads the row from the other side. If one word follows with chance 0.75, then P(any other word) = 1 − 0.75 = 0.25: one time in four the model's favorite would be the wrong guess. A word that never followed in the log gets a count of 0, and so a chance of exactly 0. Predict both numbers the code below prints before you answer.
Quick check
What does this print?
import numpy as np
counts = np.array([3, 1, 0])
p = counts / counts.sum()
print(p.tolist(), float(1 - p[0]))
[0.75, 0.25] 0.25
[3, 1, 0] 0.25
[0.75, 0.25, 0.0] 0.25
[0.75, 0.25, 0.0] 0.75
Show the answer
[0.75, 0.25, 0.0] 0.25
Dividing by the total, 4, gives 0.75, 0.25 and 0; the complement of the first is 1 − 0.75 = 0.25.
Drill
The distribution after a word
Write next_dist(log, word) that returns two lists: the words that follow word, and their probabilities. Then set names and probs for the word "was" in the log below.
log = "the dog was wet the cat was dry the dog was wet again".split()
defnext_dist(log, word):
return [], []
names, probs = next_dist(log, "was")
print(names, probs)
In the lesson, your code is checked as you write it.
In your own words
A bigram model has never seen "the stove" in its log. What probability does it give "stove" after "the", and why is that a problem?
Write a sentence or two, then compare it with a good answer in the lesson.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write