Lesson 102 turned one score into the chance of a yes. Most questions have more than two answers: which of three sign types is this, which of 50,000 words comes next. A model produces one score per answer, any numbers at all, negative or huge, and needs a fair way to turn them into a distribution: every chance positive, all of them adding up to 1, and a bigger score always getting a bigger share. Softmax is that way, and it sits at the very end of nearly every classifier and every language model. Its two practical details matter as much as the formula: the same scores can be "cooled" or "heated" with a temperature, which is the setting a chatbot exposes to make its replies safer or more varied, and a naive implementation overflows on scores a real model produces every day.
What you'll be able to do
Turn any list of scores into chances that add up to 1: exponentiate, then divide by the total
See that adding the same number to every score changes nothing, and that temperature sharpens or flattens the result
Compute softmax with a loop, then in NumPy, subtracting the largest score first so nothing overflows
Read softmax as the last step of every classifier and language model, pick the predicted class with argmax, and run it in PyTorch with F.softmax, torch.logsumexp and F.log_softmax
Quick check
Have a guess before you start. Three scores, 2, 1 and 0.1, must become three chances adding up to 1. Why not just divide each by their total, 3.1?
It would change which score is biggest
Negative scores would give negative chances
Dividing is too slow
The total of scores is always 0
Show the answer
Negative scores would give negative chances
Scores can be negative, and a score of −1 would get a share of −1/3.1. Exponentiating first makes every value positive.
The math: exponentiate, then share
Purpose. A fair way to turn any list of scores into chances that add up to 1.
Two steps. First make every score positive with the exponential: eᶻ is always above 0, and bigger z gives bigger eᶻ (lesson 24). Then divide each by their total, as lesson 88 turned counts into shares. A short table of e-values does the hand work:
z
−1
0
0.1
1
2
3
eᶻ
0.368
1
1.105
2.718
7.389
20.09
For the scores 2, 1 and 0.1: e² + e¹ + e⁰·¹ = 7.389 + 2.718 + 1.105 = 11.21. The shares are 7.389 ÷ 11.21 = 0.659, 2.718 ÷ 11.21 = 0.242 and 1.105 ÷ 11.21 = 0.099. They're all positive, they add up to 1, and the order of the scores is kept.
Quick check
From lesson 24: what is e^(a + b) in terms of eᵃ and eᵇ?
eᵃ × eᵇ
eᵃ + eᵇ
eᵃ − eᵇ
(eᵃ)ᵇ
Show the answer
eᵃ × eᵇ
Adding in the exponent multiplies the powers. That rule is why adding a constant to every score leaves softmax unchanged.
Shifting changes nothing. Add 10 to every score: each eᶻ becomes eᶻ × e¹⁰, and so does the total, so the e¹⁰ cancels in every share. Only the differences between scores matter. The shares are also the weights of a weighted average, the idea of lesson 27: weights that add to 1.
The picture. Each class is a bar, at first as tall as its score. Step by step, each bar grows into its exponential, and a last bar, the total, rises as tall as all of them stacked. Then each bar is divided by the total. Watch what that does to the bars' heights. Step through the three scores, then try the same scores with 10 added to each.
Try it now
Then set the scores to 12, 11 and 10.1 and press Use these numbers: every exponential is e¹⁰ times bigger, yet the bars keep the same shape and the shares come out exactly the same.
Symbols and formula. For scores z₁ to zₖ and a temperature T (normally 1):
pi=∑jezj/Tezi/Tsoftmax(z+c)=softmax(z)
In plain words: exponentiate each score and divide by the total; adding the same number to every score changes nothing; dividing the scores by T first flattens the chances (T above 1) or sharpens them (T below 1).
Temperature. For the scores 2, 1 and 0.1, T = 0.5 gives 0.864, 0.117 and 0.019: the winner takes almost everything. T = 2 gives 0.502, 0.304 and 0.194: much more even. As T approaches 0 softmax becomes argmax, all the chance on the top score; as T grows, the chances approach equal shares.
Quick check
Softmax of the scores 5, 5 and 5?
1/3 each
It's undefined
e⁵ each
5/15 for the first, 0 for the others
Show the answer
1/3 each
Equal scores give equal exponentials, so each gets a third. Shifting them all to 0 gives the same answer.
In your own words
In your own words: why does adding the same number to every score leave softmax unchanged?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: a loop, then NumPy, then the safe version
Purpose. Compute softmax by hand, in one line, and in the form real libraries use.
import math
import numpy as np
scores = [2.0, 1.0, 0.1]
total = 0for s in scores:
total = total + math.exp(s)
shares = []
for s in scores:
shares.append(round(math.exp(s) / total, 4))
print(shares)
z = np.array(scores)
e = np.exp(z - z.max())
p = e / e.sum()
print(p.round(4).tolist(), int(np.argmax(p)))
It runs right here in your browser, with nothing to install.
The loop and the array version agree: 0.659, 0.2424 and 0.0986. np.argmax(p) gives the position of the largest chance, the predicted class: here 0, the first. Softmax never changes which score wins; argmax of the chances is argmax of the scores.
Code flow. Here is the loop once more, written the way libraries write it: the largest score is taken away from every score before anything is exponentiated. That's the z - z.max() of the array version. Step through it and watch the boxes of exps and shares fill one class at a time, as the flow's cells did. Look closely at the numbers that land in exps.
A named mistake: exponentiating raw scores. A language model's scores can reach the hundreds. e¹⁰⁰⁰ is far above the largest float, so np.exp(1000) returns inf, and inf ÷ inf is nan: softmax of 1000 and 999 comes out [nan, nan]. Subtracting the largest score first turns them into 0 and −1, whose exponentials are 1 and 0.368, and the answer, 0.731 and 0.269, is exact, because shifting changes nothing. Every library does this; your own code should too.
Worked example
Worked example: three temperatures
Compute the softmax of the same scores at temperatures 0.5, 1 and 2.
import numpy as np
defsoftmax(z, t):
z = z / t
e = np.exp(z - z.max())
return e / e.sum()
z = np.array([2.0, 1.0, 0.1])
for t in [0.5, 1.0, 2.0]:
print(t, softmax(z, t).round(3).tolist())
In the lesson you can change this and run it.
At 0.5 the top class takes 86% of the chance; at 1 it takes 66%; at 2 only 50%, with the others catching up. The ranking never changes. That's the dial a chat model's "temperature" setting turns: low values make it pick the likeliest word almost every time, safe and repetitive; high values spread the chances, varied and more error-prone. Dividing the scores by T before the safe softmax is all it takes.
Quick check
What does this print?
import numpy as np
z = np.array([1000.0, 999.0])
e = np.exp(z - z.max())
print((e / e.sum()).round(3).tolist())
[nan, nan]
[1.0, 0.0]
[0.5, 0.5]
[0.731, 0.269]
Show the answer
[0.731, 0.269]
Shifted, the scores are 0 and −1: e⁰ = 1 and e⁻¹ = 0.368, so the shares are 0.731 and 0.269, exactly as for the scores 1 and 0.
Drill
A safe softmax
Fix softmax(z, t) so it divides the scores by t, subtracts the largest, exponentiates and shares out. Then set cool, warm and hot to the softmax of z at t = 0.5, 1 and 2, each rounded to 3 places as a list, and big to the softmax of [1000, 999] at t = 1, rounded to 3.
import numpy as np
defsoftmax(z, t):
e = np.exp(z)
return e / e.sum()
z = np.array([2.0, 1.0, 0.1])
cool = softmax(z, 0.5).round(3).tolist()
warm = softmax(z, 1.0).round(3).tolist()
hot = softmax(z, 2.0).round(3).tolist()
big = softmax(np.array([1000.0, 999.0]), 1.0).round(3).tolist()
print(cool, warm, hot, big)
In the lesson, your code is checked as you write it.
In your own words
A chat app offers a temperature setting from 0 to 2. What would you expect its replies to look like at 0.2 and at 1.8?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: the last step of every classifier
Purpose. See softmax turn a model's final scores into its answer.
Classifiers. A sign classifier's last layer (lesson 56) produces one score per class: arrow, tent and trail. Softmax turns the three scores into three chances, the predicted class is the argmax, and its chance is how sure the model is. For four pictures with scores (3, 1, 0.2), (0.5, 2.5, 1), (1, 1.2, 0.9) and (0.1, 0.3, 4), the predictions are arrow with 0.84, tent with 0.74, tent with only 0.39, and trail with 0.96. The third picture is a near tie: worth a second look, or a human check.
Language models. The same step, wider: a language model scores every word in its vocabulary, tens of thousands of them, and softmax turns those scores into the next-word distribution of lesson 91, which is then sampled with a temperature. Here are the twelve scores the camp-log model of lesson 65 gave the words of its vocabulary after "the fire is", shared out at temperatures 1 and 0.5. Each line prints the temperature, the likeliest word and its chance, the chance of "fire", and the total:
import numpy as np
vocab = ["by", "fire", "is", "lake", "lit", "on", "rain", "the", "tent", "trail", "up", "wet"]
scores = np.array([0.31, 1.01, 0.42, -0.96, 1.1, 0.36, -0.96, 0.4, -0.27, -0.18, 0.41, -0.94])
for t in [1.0, 0.5]:
z = scores / t
e = np.exp(z - z.max())
p = e / e.sum()
best = int(np.argmax(p))
print(t, vocab[best], round(float(p[best]), 3), round(float(p[1]), 3), round(float(p.sum()), 3))
At temperature 1, "lit" is the likeliest next word, yet it gets only 0.19, with "fire" close behind at 0.174: twelve words share the chance, and a sampled reply could easily say either. At 0.5 the gaps between scores double, and "lit" climbs to 0.311. Twelve scores or fifty thousand, the code is identical. For a whole batch at once, the softmax runs along each row: subtract each row's largest score, exponentiate, and divide by each row's total, with axis=1 and keepdims=True so every row is shared out on its own.
Quick check
A classifier's softmax gives 0.39, 0.32 and 0.29 for one picture. What should you take from it?
It's 39% sure, so the answer is definitely wrong
It barely prefers the first class: a near tie worth checking
The scores must have been negative
The model is broken
Show the answer
It barely prefers the first class: a near tie worth checking
The argmax is the first class, but the chances are close: the model is unsure about this picture.
Quick check
What does this print?
import numpy as np
S = np.array([[1.0, 0.0], [0.0, 0.0]])
E = np.exp(S - S.max(axis=1, keepdims=True))
print((E / E.sum(axis=1, keepdims=True)).round(3).tolist())
[[0.576, 0.212], [0.212, 0.212]]
[[0.731, 0.269], [0.5, 0.5]]
[[0.731, 0.269], [0.731, 0.269]]
[[1.0, 0.0], [0.5, 0.5]]
Show the answer
[[0.731, 0.269], [0.5, 0.5]]
Each row is shared out on its own: scores 1 and 0 give 0.731 and 0.269, and the tie gives 0.5 each.
In PyTorch. The real tool has softmax built in. F.softmax(z, dim=-1) shares out along the last axis, which for a batch means each row on its own, as axis=1 did. Two relatives come with it. torch.logsumexp(z, dim=-1) is the natural log of the total, ln Σ eᶻ, worked out the safe way: the largest score plus the log of the shifted total, so nothing overflows. And F.log_softmax gives the log of every share, as each score minus that log-total:
import torch
import torch.nn.functional as F
z = torch.tensor([2.0, 1.0, 0.1])
print(F.softmax(z, dim=-1))
print(torch.logsumexp(z, dim=-1), torch.log(torch.exp(z).sum()))
print(F.log_softmax(z, dim=-1))
print(z - torch.logsumexp(z, dim=-1))
The first line is the NumPy cell's 0.659, 0.2424 and 0.0986, written to 4 places as tensors print. The log of the total is 2.417 both ways: e to the 2.417 is the flow's total, 11.21, and 2.417 is also 2 plus ln 1.5174, the stepper's shifted total. The log-shares, −0.417, −1.417 and −2.317, are each score minus 2.417. The next lesson builds its loss from exactly these: −log of the right answer's chance. Why not just take the log of the softmax? Predict what each line gives for scores 1000 apart.
Quick check
What does this print?
import torch
import torch.nn.functional as F
z = torch.tensor([1000.0, 0.0])
print(F.log_softmax(z, dim=-1).tolist(), torch.log(F.softmax(z, dim=-1)).tolist())
[0.0, -inf] [0.0, -inf]
[nan, nan] [nan, nan]
[0.0, -1000.0] [0.0, -1000.0]
[0.0, -1000.0] [0.0, -inf]
Show the answer
[0.0, -1000.0] [0.0, -inf]
F.log_softmax works out each log-share as score minus log-total: 0 − 1000 = −1000, a perfectly good number. The second line first makes the share e⁻¹⁰⁰⁰, far too small for a float, so it becomes 0, and the log of 0 is −inf. A loss built on that would be infinite, which is why losses start from log_softmax.
Drill
Predictions for four pictures
Each row of S holds one picture's scores for arrow, tent and trail. Set P to the row-by-row softmax (subtract each row's max, use axis=1, keepdims=True), pred to the predicted class of each picture as a list, and sure to each prediction's chance, rounded to 3, as a list.
import numpy as np
S = np.array([[3.0, 1.0, 0.2], [0.5, 2.5, 1.0], [1.0, 1.2, 0.9], [0.1, 0.3, 4.0]])
P = S
pred = []
sure = []
print(pred, sure)
In the lesson, your code is checked as you write it.
In your own words
Softmax always gives every class some chance, even a class the model has no reason to think likely. Why is that useful for a language model?
Write a sentence or two, then compare it with a good answer in the lesson.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write