How big is a list of numbers? There's more than one fair answer. The straight-line length of its arrow is one, the L2 norm. The distance you'd walk along a grid of streets is another, the L1 norm. And sometimes only the biggest number matters, the max norm. Machine learning uses all three: to measure errors, to keep weights small, and to stop a training step from growing too large. This lesson puts them side by side, computes each with a loop and then with one NumPy call, turns any vector into a unit vector of length 1, and shows where each norm turns up inside the losses and safety limits of real training code.
What you'll be able to do
Measure a vector's size three ways: the L2 norm, the L1 norm and the max norm
Make a unit vector by normalizing, and find a distance as the norm of a difference
Compute norms with a loop and with np.linalg.norm, and check that a unit vector has length 1
See the norms inside ML: mean absolute and squared error, weight decay, and gradient clipping, in NumPy and in PyTorch
Quick check
Have a guess before you start. To get from (0, 0) to (3, 4), how far do you walk on a grid of streets, and how far does a bird fly?
Both 5
Walk 5, fly 7
Both 7
Walk 7, fly 5
Show the answer
Walk 7, fly 5
On streets you go 3 across and 4 up: 7 blocks. The bird flies straight: √(3² + 4²) = 5. Two honest ways to measure one movement.
The math: how big is a vector?
Purpose. One number for the size of a whole list of numbers.
A norm measures a vector's size. Three are common:
The L2 norm, ‖v‖₂: the arrow's straight-line length, Pythagoras in any number of dimensions. Square each component, add, take the root. For (3, −4), that's 5.
The L1 norm, ‖v‖₁: add the sizes of the components, ignoring signs. For (3, −4), that's 7: the street-grid walk.
The max norm, ‖v‖∞: the biggest size of any one component. For (3, −4), that's 4.
When people say "the norm" or "the length" with no label, they mean L2.
Quick check
From an earlier lesson: what is |−6|?
−6
0
36
6
Show the answer
6
The absolute value is the distance from zero, so |−6| = 6. The L1 norm adds these sizes.
The picture. Watch the L2 norm of (3, −4): square each component, add the squares, then take the square root. The picture draws the arrow with its two legs, 3 across and 4 down, and builds a real square on each leg: 3² is a 3 by 3 square of 9 small squares. Add the two areas, take the root, and you have the arrow's length.
Try it now
Then set which norm to 1 and press Use these numbers: the picture walks the legs like streets, 3 + 4 = 7. Set it to max for the biggest size. A vector of three components, like 1, 2, 2, works too (length 3), but has no flat picture.
Symbols and formula. The three norms, and a unit vector:
∥v∥2=∑vi2∥v∥1=∑∣vi∣v^=∥v∥v
In plain words: L2 is the root of the sum of squares; L1 is the sum of sizes. Dividing a vector by its length makes a unit vector, v̂ ("v hat"), with length exactly 1 and the same direction.
Distance is a norm. The distance between points a and b is the length of the arrow between them: ‖b − a‖₂. That's lesson 41's distance formula, now in one symbol. Making a unit vector is called normalizing: (3, −4) normalized is (0.6, −0.8).
Which norm when? All three give 0 for the zero vector, and doubling v doubles each of them. They differ in what they count. L2 is the length you'd measure with a ruler, and it punishes one big component more than several small ones. L1 counts every component's size equally, as a walk along a grid would. The max norm cares only about the worst component, which suits limits like "no weight may exceed 5". For (3, −4) they give 5, 7 and 4: the L1 norm is never smaller than the other two, and the max norm never bigger.
Three shapes of size 1. Why does L1 always come out biggest? Draw every arrow of size 1 under each norm: a circle for L2, a diamond for L1, a square for max. Press Next to see each one with v = (0.6, 0.8), then all three.
Quick check
What is the L2 norm of (1, 2, 2)?
√5
5
3
9
Show the answer
3
1 + 4 + 4 = 9, and √9 = 3.
Quick check
What is (6, 8) normalized?
(6, 8)
(0.6, 0.8)
(0.75, 1)
(3, 4)
Show the answer
(0.6, 0.8)
Its length is √(36 + 64) = 10. Dividing each component by 10 gives (0.6, 0.8), length 1.
In your own words
In your own words: what does each of the three norms measure about a vector?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: loops, then np.linalg.norm
Purpose. Write each norm once by hand to see it, then use NumPy's one call.
import math
defl1(v):
total = 0for x in v:
total = total + abs(x)
return total
defl2(v):
total = 0for x in v:
total = total + x ** 2return math.sqrt(total)
print(l1([3, -4]), l2([3, -4]), l2([1, 2, 2]))
It runs right here in your browser, with nothing to install.
Code flow. Here are both loops folded into one, on (3, −4), with NumPy's answer at the end. Step through it and watch the two running totals: walk adds each size, and total adds each square, 9 then 16, the same squares the flow in the math part wrote in its boxes.
NumPy has all three in np.linalg.norm, chosen with ord. With no ord it's L2:
import numpy as np
v = np.array([3, -4])
print(np.linalg.norm(v), np.linalg.norm(v, ord=1), np.linalg.norm(v, ord=np.inf))
np.inf is infinity, NumPy's name for the max norm.
Normalizing, and checking it. Divide by the norm, then confirm the result has length 1:
import numpy as np
v = np.array([3, -4])
u = v / np.linalg.norm(v)
print(u.tolist(), np.linalg.norm(u))
A named mistake: normalizing the zero vector. (0, 0) has length 0, and dividing by 0 gives nan ("not a number") with a warning. The zero vector has no direction to keep, so code that normalizes should check for a zero length first.
Worked example
Worked example: three distances between the same two points
Measure the gap from (1, 2) to (4, 6) with each norm of the difference.
import numpy as np
a = np.array([1, 2])
b = np.array([4, 6])
d = b - a
print(d.tolist())
print(np.linalg.norm(d), np.linalg.norm(d, ord=1), np.linalg.norm(d, ord=np.inf))
In the lesson you can change this and run it.
The difference is (3, 4). Its L2 norm, 5.0, is the straight-line distance. Its L1 norm, 7.0, is the street walk. Its max norm, 4.0, is the biggest single gap, across or up. Each is a fair answer to "how far apart?", and which one a model uses changes what it treats as close.
Quick check
What does this print?
import numpy as np
v = np.array([2, -2, 1])
print(np.linalg.norm(v), np.linalg.norm(v, ord=1))
9.0 5.0
5.0 3.0
3.0 5.0
3.0 1.0
Show the answer
3.0 5.0
L2: √(4 + 4 + 1) = 3.0. L1: 2 + 2 + 1 = 5.0.
Quick check
Which line makes v a unit vector?
v - np.linalg.norm(v)
v * np.linalg.norm(v)
v / v.max()
v / np.linalg.norm(v)
Show the answer
v / np.linalg.norm(v)
Dividing by the length scales it to length 1, keeping its direction.
Drill
Two norms and a unit vector
Fix l1 and l2, then set u to (6, −8) normalized, as a list.
import math
import numpy as np
defl1(v):
total = 0for x in v:
total = total + x
return total
defl2(v):
total = 0for x in v:
total = total + x ** 2return total
u = (np.array([6, -8]) / 8).tolist()
print(l1([3, -4]), l2([3, -4]), u)
In the lesson, your code is checked as you write it.
In your own words
Why can't the zero vector be normalized?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: errors, penalties and limits
Purpose. Norms appear wherever a model needs one number for the size of many.
Errors. For predictions ŷ and answers y, the error vector is ŷ − y. Its L1 norm, divided by n, is the mean absolute error; its L2 norm squared, divided by n, is the mean squared error from lesson 26:
MAE=n1∥y^−y∥1MSE=n1∥y^−y∥22
In plain words: MAE averages the sizes of the misses; MSE averages their squares, so it punishes big misses more.
import numpy as np
pred = np.array([3.0, 5.0, 9.0])
y = np.array([2.0, 5.0, 7.0])
err = pred - y
print(np.abs(err).mean(), (err ** 2).mean())
Quick check
What does this print?
import numpy as np
w = np.array([0.5, -1.0, 2.0])
print(np.sum(w ** 2))
5.25
1.5
2.29
3.5
Show the answer
5.25
0.25 + 1 + 4 = 5.25: the L2 norm squared, the size weight decay penalizes.
Weight decay adds a small multiple of ‖w‖₂² to the loss, so training prefers small weights, which tend to generalize better. Gradient clipping caps a step: if ‖g‖ is bigger than a limit, g is scaled down to exactly that length, keeping its direction. For a limit of 1, that's just normalizing:
import numpy as np
g = np.array([3.0, 4.0])
n = np.linalg.norm(g)
if n > 1:
g = g / n
print(n, g.tolist(), np.linalg.norm(g))
A gradient of length 5 becomes (0.6, 0.8), length 1: same direction, a safe size.
In PyTorch. The norm, clipping and weight decay are all built in. torch.linalg.norm(g) is the L2 norm. nn.utils.clip_grad_norm_ clips: it takes the weights whose gradients to clip and the limit, max_norm, and shrinks their gradients in place (the name ends in _). Weight decay is an option on the optimizer, weight_decay=. As in lesson 48, w.grad holds each gradient, put there by hand:
import torch
import torch.nn as nn
g = torch.tensor([3.0, 4.0])
print(torch.linalg.norm(g))
w = torch.tensor([1.0, 1.0], requires_grad=True)
w.grad = torch.tensor([3.0, 4.0])
nn.utils.clip_grad_norm_([w], max_norm=1.0)
print(w.grad)
w = torch.tensor([0.5, -1.0, 2.0], requires_grad=True)
w.grad = torch.zeros(3)
opt = torch.optim.SGD([w], lr=0.1, weight_decay=0.1)
opt.step()
print(w)
The first two lines match the NumPy cell: the gradient's length, 5, then the clipped gradient (0.6, 0.8), in PyTorch's four decimals. Clipping looks only at the gradient, so the weights' own values don't matter there. The last line is weight decay on its own. The gradient is all zeros (torch.zeros(3) makes three), yet the step still moves every weight toward 0: weight_decay=0.1 adds 0.1 × w to the gradient, which is how the ‖w‖₂² penalty in the loss reaches each weight. With lr=0.1, each weight loses 0.1 × 0.1 = 0.01 of itself, so (0.5, −1, 2) becomes (0.495, −0.99, 1.98).
Quick check
What does this print? The limit is 5 this time.
import torch
import torch.nn as nn
w = torch.tensor([0.0, 0.0], requires_grad=True)
w.grad = torch.tensor([6.0, 8.0])
nn.utils.clip_grad_norm_([w], max_norm=5.0)
print(w.grad)
tensor([3.0000, 4.0000])
tensor([0.6000, 0.8000])
tensor([5., 5.])
tensor([6., 8.])
Show the answer
tensor([3.0000, 4.0000])
(6, 8) has length 10, over the limit of 5, so it's scaled by 5 ÷ 10 = 0.5 to (3, 4): length 5, same direction.
Why squares in the loss, and absolute values elsewhere? Squaring errors (MSE) makes big mistakes count far more than small ones, which pushes a model hard to fix its worst predictions. Absolute errors (MAE) treat every unit of error the same, so one wild example can't dominate. Neither is "right": the choice says which mistakes matter most for the job at hand. A model that predicts house prices might use MAE so that one mansion doesn't dominate, while one that must never be far wrong, like a dose calculator, might use MSE so that big misses are punished hardest.
Drill
Clip a gradient, measure two errors
Fix the code: clip(g) should scale g down to length 1 when its length is above 1, and leave a shorter g as it is. Then make mae and mse the mean absolute and mean squared errors.
import numpy as np
defclip(g):
n = np.linalg.norm(g)
if n < 1:
g = g / 10return g
pred = np.array([1.0, 4.0, 4.0])
y = np.array([2.0, 4.0, 1.0])
err = pred - y
mae = err.mean()
mse = (err * 2).mean()
print(clip(np.array([6.0, 8.0])).tolist(), mae, mse)
In the lesson, your code is checked as you write it.
In your own words
Why might a model be trained with MSE instead of MAE, or the other way around?
Write a sentence or two, then compare it with a good answer in the lesson.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write