A logarithm answers one question: to what power? log₁₀(1,000) asks "10 to what power gives 1,000?", and the answer is 3. Logs undo exponentials, and they turn multiplying into adding, which is why they're everywhere in machine learning. Multiply a hundred small probabilities and a computer rounds the answer to zero; add their logs and nothing is lost. The loss most models are trained on is built from a log. This lesson defines logs by that question first, then the three laws that follow from it, and the one mistake almost everyone makes with them.
What you'll be able to do
Read a logarithm as the answer to "to what power?", in base 10, base 2 and base e
Use the three laws of logs, and solve an equation like 2ˣ = 1,000
Compute logs in Python with math.log, math.log10 and math.log2, and test each law with numbers
Add logs instead of multiplying tiny probabilities, and read −log(p) as a measure of surprise
Quick check
Have a guess before you start. 10 to what power gives 1,000?
10
100
30
3
Show the answer
3
10 × 10 × 10 = 1,000, so the power is 3. That's exactly what log₁₀(1,000) means.
The math: to what power?
Purpose. Undo an exponential, and tame numbers of hugely different sizes.
A logarithm answers "to what power?":
loga(y)=xmeansax=y
In plain words: log base a of y is the power you raise a to, to get y.
So log₁₀(100) = 2, because 10² = 100, and log₂(8) = 3, because 2³ = 8. Three bases are common: log₁₀ (base 10, "orders of magnitude" from lesson 11), log₂ (base 2, doublings), and ln, the natural log, base e. log of 1 is 0 in every base, since anything to the power 0 is 1.
Quick check
From an earlier lesson: what is 2¹⁰?
512
100
1,024
20
Show the answer
1,024
Ten 2s multiplied make 1,024. Asked the other way around, log₂(1,024) = 10.
The picture. Asking the question the other way around has a picture too. Below is the curve y = 2ˣ from lesson 24, through (1, 2), (2, 4) and (3, 8): each step right doubles the height. Press Next to swap the two numbers in every point. (3, 8) becomes (8, 3), because log₂(8) = 3, and the swapped points draw the curve y = log₂ x, the exponential's mirror image across the dashed line y = x.
Try it now
Logs turn multiplying into adding. Since 2³ × 2² = 2⁵, multiplying two powers of 2 adds their exponents. Logs are exponents, so the log of a product is the sum of the logs. Watch 2 × 8 × 4 done with base-2 logs: add 1, 3 and 2 to get 6, then 2⁶ = 64. The flow's picture is a number line on a log scale, marked in powers of 2: the tick marked 3 stands for 2³ = 8, so every number sits at its own log. Multiplying by 8 is then a jump of 3 ticks, and the jumps add up.
Symbols and formula. The three laws, each the law of exponents seen backward:
log(ab)=loga+logblogba=loga−logblog(an)=nloga
In plain words: a product's log is a sum, a division's is a difference, and a power comes out in front.
Solving with logs. To solve 2ˣ = 1,000, ask "2 to what power?": x = log₂(1,000) ≈ 9.97. Doubling about ten times passes a thousand, as you've seen before. Any base works if you divide: x = log(1,000) ÷ log(2).
In your own words: why does the log of a product equal the sum of the logs?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: math.log, and testing the laws
Purpose. Python computes logs in every common base, and checks each law with numbers.
import math
print(math.log10(100), math.log2(8), math.log(math.e), math.log(1))
print(math.log(math.exp(3)))
It runs right here in your browser, with nothing to install.
math.log on its own is the natural log, base e. math.log10 and math.log2 are bases 10 and 2. The last line shows that log undoes exp: the natural log of e³ is 3.0.
Code flow. Each law can be tested by trying numbers. Step through a test of all three on 8 and 4, two of the numbers from the flow. la and lb hold their logs, and each print line puts the two sides of one law next to each other.
A named mistake: log(a + b). The laws are about products, not sums. log(a + b) is not log a + log b, and one line shows it:
import math
print(math.log10(2 + 8), math.log10(2) + math.log10(8))
The first is log₁₀(10) = 1.0, while the sum of logs is log₁₀(16) ≈ 1.20. A sum inside a log stays stuck.
Another: log of 0. No power of 10 gives 0, so log(0) has no answer, and math.log10(0) stops with an error. Code that takes logs of probabilities has to make sure none of them is exactly 0, a problem lesson 105 returns to.
Worked example
Worked example: how many doublings?
How many doublings take 1 past a million? And how many years at 7% growth double an amount?
import math
print(math.log2(1000000))
print(math.log(1000000) / math.log(2))
print(round(math.log(2) / math.log(1.07), 2))
In the lesson you can change this and run it.
log₂(1,000,000) ≈ 19.93, so 20 doublings pass a million. The second line gets the same answer from natural logs, dividing log(1,000,000) by log(2): that's how to find a log in any base. The last line solves 1.07ˣ = 2, the doubling time at 7%: about 10.24 years, so it passes double in the 11th year, as lesson 24's loop found.
Quick check
What does this print?
import math
print(math.log10(1000), math.log2(1))
1000.0 1.0
3.0 0.0
3.0 1.0
3 1
Show the answer
3.0 0.0
10³ = 1,000, so log₁₀ is 3.0. Anything to the power 0 is 1, so log₂(1) is 0.0. Logs come back as floats.
Quick check
Which line solves 3ˣ = 50?
50 / 3
math.log(50 / 3)
math.log(50) / math.log(3)
math.log(50) - math.log(3)
Show the answer
math.log(50) / math.log(3)
x = log₃(50), and any base's log is the natural log divided by the natural log of the base.
Changing base. Python has three log functions built in, but any base works through one division: log_a(y) = log(y) ÷ log(a), using the same log on top and bottom. That's how the worked example found a base-1.07 log. It also shows the bases are one idea at different scales: log₂ of a number is always about 3.32 times its log₁₀, because log₂(10) ≈ 3.32.
Drill
Test the laws, and solve
Fix the code: product_way and sum_way should both be log₁₀(4 × 25), and x should solve 2ˣ = 4,096.
import math
product_way = math.log10(4 + 25)
sum_way = math.log10(4) * math.log10(25)
x = 4096 / 2print(product_way, sum_way, x)
In the lesson, your code is checked as you write it.
In your own words
Why is log(a + b) not equal to log a + log b?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: adding logs instead of multiplying probabilities
Purpose. A model's probabilities multiply, and tiny numbers multiplied vanish. Logs keep them safe.
A language model gives each word of a sentence a probability, and the sentence's probability is all of them multiplied. With 120 words at 0.001 each, the true answer is 10⁻³⁶⁰, far smaller than a float can hold, so Python rounds it to 0.0. Adding the logs gives −360, a perfectly ordinary number:
log(p1p2⋯pn)=logp1+logp2+⋯+logpn
In plain words: instead of multiplying many small chances, add their logs.
import math
prob = 1.0
total = 0for i inrange(120):
prob = prob * 0.001
total = total + math.log10(0.001)
print(prob, round(total, 6))
Quick check
What does this print?
import math
print(round(math.log(0.9 * 0.8), 6), round(math.log(0.9) + math.log(0.8), 6))
0.72 -0.328504
-0.328504 1.7
-0.328504 -0.328504
-0.105 -0.223
Show the answer
-0.328504 -0.328504
The log of a product equals the sum of the logs, so both are ln(0.72) ≈ −0.3285. Logs of probabilities are negative, because they're below 1.
−log(p) measures surprise. For a probability p between 0 and 1, log(p) is negative, so −log(p) is positive. It's small when p is near 1 (no surprise) and grows fast as p shrinks (a big surprise):
import math
for p in [0.9, 0.5, 0.1, 0.01, 0.001]:
print(p, round(-math.log(p), 3))
This is the core of cross-entropy, the loss most classifiers and language models are trained on: −log of the probability the model gave the right answer. A confident right answer costs little; a confident wrong one costs a lot. The picture puts each of those five chances beside its −log.
A log scale. Logs also tame ranges. Model sizes run from thousands to trillions of parameters, and on an ordinary axis the small ones are squashed into a corner. On a log scale each step up the axis is ten times the last, so a thousand, a million and a billion sit evenly spaced, at 3, 6 and 9, just as the flow's number line in the math part was marked in powers of 2. Charts of training loss and model size almost always use log scales for this reason.
Drill
Probabilities the safe way
A model gives four words the probabilities in probs. Set log_total to the sum of their natural logs (math.log), and surprise to −log of the smallest one.
import math
probs = [0.5, 0.2, 0.1, 0.05]
log_total = 1for p in probs:
log_total = log_total * math.log(p)
surprise = math.log(0.05)
print(log_total, surprise)
In the lesson, your code is checked as you write it.
In the real tools. In PyTorch, the library most models are built with, a whole list of chances is one tensor, and a single line takes every log and adds them. Here are the drill's four chances:
In your own words
Why do models add log-probabilities instead of multiplying probabilities?
Write a sentence or two, then compare it with a good answer in the lesson.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write