A power is a short way to write repeated multiplying, and a root undoes it. Powers describe how fast things grow: doubling ten times multiplies by about a thousand. In machine learning, errors are squared before they're added up, and the sums inside attention are scaled down by a square root. This lesson builds the three laws of powers from written-out factors, so you can see why they're true instead of memorizing them.
What you'll be able to do
Read a power as repeated multiplying, and use the three laws of powers
Explain why a⁰ = 1 and a⁻ⁿ = 1/aⁿ, and turn roots into powers
Compute powers and roots in Python with ** and math.sqrt, and improve a guess for a square root
Work out squared errors and the 1/√d scale used in attention
Quick check
Have a guess before you start. Which is bigger, 2¹⁰ or 10²?
10²
2¹⁰
They're equal
Can't tell without a calculator
Show the answer
2¹⁰
2¹⁰ is ten 2s multiplied: 1,024. 10² is two 10s multiplied: 100. Ten doublings beat two tens by far.
The math: repeated multiplying
Purpose. Multiplying the same number again and again is common enough to get its own short form.
25=2×2×2×2×2=32
In plain words: the big number, 2, is the base. The small raised number, 5, is the exponent: it counts how many 2s are multiplied.
Read 2⁵ as "2 to the power 5". A power of 2 is called squared (side × side for a square) and a power of 3 cubed (side × side × side for a cube). The picture draws both, then shows what each extra factor does.
Try it now
A named mistake: 4² is not 4 × 2. 4² is two 4s multiplied, 4 × 4 = 16. Doubling 4 gives 8, which is a different job. The exponent counts factors; it never multiplies the base.
Quick check
From an earlier lesson: what is (−3) × (−3)?
0
−9
−6
9
Show the answer
9
Negative times negative is positive: (−3) × (−3) = 9. So (−3)² = 9, a fact that matters in Python later in this lesson.
The laws come from the factors. Write powers out as factors and three rules fall out. Multiply two powers of the same base and the factors join into one longer row. Watch 2³ × 2⁴. The cells show the factors; the picture shows how big the number gets, in squares. Each factor of 2 doubles the squares, so every power of 2 is twice the size of the one before.
Dividing cancels factors in pairs. On top and bottom of a division, each matching pair of factors divides to 1 and drops out. That's where two surprising facts come from. Watch 3³ ÷ 3³: every factor finds a partner. In the picture, 3³ is 27 squares, and each ÷ 3 on the bottom keeps one third of them.
Symbols and formula. The three laws, for any base a:
aman=am+nanam=am−n(am)n=amn
In plain words: multiply, add the exponents. Divide, take them away. A power of a power, multiply them.
The third law holds because (a^m)^n is n copies of a row of m factors, so m × n factors in all.
When the bottom has more factors, some are left there: 2¹ ÷ 2³ = 1 ÷ 2². The law gives 2⁻², so a negative exponent means one over: 2⁻² = 1/4 = 0.25. And when everything cancels, a⁰ = 1, for any base except 0.
Quick check
What is 5³ × 5² as one power of 5?
10⁵
5⁵
5⁶
25⁵
Show the answer
5⁵
Same base, so add the exponents: 3 + 2 = 5, giving 5⁵.
Quick check
What is 2⁻²?
−0.25
0.25
−4
4
Show the answer
0.25
A negative exponent means one over: 2⁻² = 1 ÷ 2² = 1/4 = 0.25.
Roots undo powers. The square root √a is the number that, squared, gives a: √25 = 5, since 5² = 25. The cube root ∛27 = 3, since 3³ = 27. Roots are powers too:
a1/2=a
In plain words: to the power one half is the square root, because squaring it gives back a.
That's the third law at work: (a^(1/2))² = a^(1/2 × 2) = a¹ = a.
A second named mistake: √a + √b is not √(a + b). √9 + √16 = 3 + 4 = 7, but √(9 + 16) = √25 = 5. Roots don't split over a sum.
Quick check
What is √36 + √64?
14
100
√100
10
Show the answer
14
Take each root first: √36 = 6 and √64 = 8, so the sum is 14.
In your own words
In your own words: why is any number (except 0) to the power 0 equal to 1?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: ** and square roots
Purpose. Python writes a power with two stars, **. Square roots come from math.sqrt or from the power 0.5.
It runs right here in your browser, with nothing to install.
A negative or decimal exponent gives a float. math.sqrt always gives a float too, even for a perfect square.
Code flow. A power is repeated multiplying, so doubling a number three times, one line at a time, must land where 2 ** 3 does in one line. Step through and watch size double.
A named mistake: ^ isn't a power. In Python, ^ does a different job on whole numbers (it compares their binary digits). So 2 ^ 3 runs without any error and quietly gives the wrong answer:
print(2 ^ 3) # not a powerprint(2 ** 3) # the power
Minus signs and powers.** binds tighter than a minus sign in front of it, so -5 ** 2 means −(5²) = −25. Brackets make the minus part of the base:
print(-5 ** 2)
print((-5) ** 2)
Testing a law by trying numbers. Python can check a law for any numbers you pick. It can also show a mistake is a mistake:
import math
a = 3
m = 4
n = 2print(a ** m * a ** n == a ** (m + n))
print(a ** m / a ** n == a ** (m - n))
print(math.sqrt(9) + math.sqrt(16) == math.sqrt(9 + 16))
print(4 ** 2, 4 * 2)
One true example doesn't prove a law, but one false example is enough to sink a wrong one.
Guess and improve. How could you find √10 with no root key? Guess, then improve. If g is a guess for √n, then n ÷ g lands on the other side of the root, so the average of the two is a better guess. Three rounds, written out by hand:
import math
n = 10
guess = 3
guess = (guess + n / guess) / 2print(guess)
guess = (guess + n / guess) / 2print(guess)
guess = (guess + n / guess) / 2print(guess)
print(math.sqrt(n))
By the third round the guess matches math.sqrt to many places. This is your first repeated-improvement method: start somewhere, then make the answer better a step at a time. Training a model works the same way.
Worked example
Worked example: how fast doubling grows
A dataset starts at 1,000 examples and doubles every year. How big is it after 10 years, and after 20?
Doubling 10 times multiplies by 2¹⁰ = 1,024, which is close to a thousand. So 10 doublings take 1,000 examples to about a million, and 20 doublings multiply by 2²⁰ = 1,048,576, about a million. "Ten doublings is about a thousand" is a handy rule for sizes that double, from datasets to computer memory. And ** runs before *, so each line computes the power first.
Quick check
What does this print?
print(-3 ** 2, (-3) ** 2)
-9 9
9 9
9 -9
-9 -9
Show the answer
-9 9
runs before the leading minus, so -3 2 is −(3²) = −9. With brackets, (−3)² = 9.
Quick check
Which line computes the square root of x?
x ** 2
x / 2
x ** 0.5
x ^ 0.5
Show the answer
x ** 0.5
To the power one half is the square root.
Drill
Fix the powers
Fix each line: cube should be 4 cubed, root the square root of 144, tiny 10 to the power −3, and law should check that 5² × 5³ equals 5⁵.
In the lesson, your code is checked as you write it.
In your own words
Why does each round of guess-and-improve get closer to √n?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: squares for errors, roots for scale
Purpose. Sizes in ML grow by powers, and roots tame them. Three places you'll meet them first: the loss, the scale inside attention, and memory.
Squared error. A model's miss on one example is its prediction minus the target. Squaring the miss makes every error positive, and it punishes big misses far more than small ones:
squared error=(y^−y)2
In plain words: ŷ, read "y hat", is the prediction and y is the target. Take the miss and multiply it by itself. A miss of 2 costs 4; a miss of 10 costs 100.
The miss is 4 − 6 = −2, and squaring gives (−2)² = 4. The brackets make −2 the base, so the sign disappears.
Big misses cost far more. Square four misses, two below the target and two above:
print((-3) ** 2, (-1) ** 2, 0.5 ** 2, 2 ** 2)
The picture puts each miss on the curve of squares: the miss runs across, its squared error goes up.
In the real tools. In PyTorch, the library most models are built with, a loss is worked out for a whole batch of predictions at once. Here are the four misses above, as predictions and targets. One line squares every miss and averages the squares: the mean squared error, the usual loss for a model that predicts a number.
Scaling by 1 over a root. Attention, the core step of a transformer, adds up d products, where d is the size of each vector, often 64 or 768. When the terms are random, a sum of d of them tends to grow like √d. So the result is divided by √d to keep it in a steady range:
scale=d1
In plain words: the bigger the vectors, the more the sums are shrunk, by one over the square root.
Vectors in an attention layer have size d = 256. What is the scale 1/√d?
1/16 = 0.0625
16, since √256 = 16
1/256 ≈ 0.0039
1/128 ≈ 0.0078
Show the answer
1/16 = 0.0625
√256 = 16, since 16 × 16 = 256. So the scale is 1/16 = 0.0625.
Powers of two are everywhere. Layer widths, vocabulary sizes and memory come in powers of two: 256, 512, 1,024. With 2¹⁰ ≈ 1,000 you can size them in your head.
Memory is the count of numbers times the bytes each one takes, and the first law does the sizing. A table of weights 1,024 wide and 1,024 tall holds 2¹⁰ × 2¹⁰ = 2²⁰ numbers. At 4 bytes, or 2², per number, that's 2²⁰ × 2² = 2²² bytes:
A little over 4 million bytes for one table, and a large model holds hundreds of tables, many far bigger.
Drill
Squared errors and a scale
The sign reader predicted five brightness values. Set e1 to e5 to the squared errors (prediction minus target, squared), total to their sum, and scale to 1/√d for d = 64. Four lines have a slip.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write