AI work mixes numbers of wildly different sizes: billions of parameters, learning rates of a few ten-thousandths, tiny numbers added to keep a division safe. Written out in full they're hard to read and easy to get wrong by a zero or two. Scientific notation writes each one as a small number times a power of ten, so sizes can be read, compared and multiplied at a glance. Python writes it with an e, as in 3e-4.
What you'll be able to do
Write numbers in scientific notation, a × 10ᵏ, and back again
Compare and rank numbers by order of magnitude, and know the prefixes from nano to tera
Read and write e-notation in Python, and find an order of magnitude with math.log10
Work out a model's memory from its parameter count, and compare learning rates
Quick check
Have a guess before you start. Which is bigger: 3 × 10⁻⁴ or 2 × 10⁻³?
3 × 10⁻⁴
They're equal
Can't tell without working them out
2 × 10⁻³
Show the answer
2 × 10⁻³
3 × 10⁻⁴ = 0.0003 and 2 × 10⁻³ = 0.002. The power of ten decides first: 10⁻³ is ten times 10⁻⁴, and that outweighs 3 against 2.
The math: powers of ten
Purpose. Write any number, however big or small, as a short number times a power of ten.
Powers of ten are ones with zeros: 10³ = 1,000 and 10⁶ = 1,000,000. With negative exponents they're one over: 10⁻³ = 1/1,000 = 0.001.
A number in scientific notation is written a × 10ᵏ, where a is at least 1 and less than 10:
4,500,000=4.5×1060.00034=3.4×10−4
In plain words: move the decimal point until one digit that isn't zero sits in front of it. The power of ten counts the moves: positive for a big number, negative for a small one.
Quick check
From the last lesson: what is 10⁻²?
−20
0.2
−100
0.01
Show the answer
0.01
A negative exponent means one over: 10⁻² = 1/10² = 1/100 = 0.01.
Watch 0.00034 become scientific notation, one move of the point at a time. Each move right times the number by 10, so the power of ten drops by one to keep the value the same.
The picture. The number line under the formula isn't spaced like a ruler. Each tick is a power of ten, ten times the tick before it, from 10⁻⁴ on the left up to 10¹. So every move of the point is one equal jump: one tick to the right, × 10. The lit dot is the front number, the one before the × in the cells. The faint dot it leaves behind is 0.00034 itself, which never moves, because the power of ten makes up for each jump. The shaded band runs from 1 up to (not including) 10. The front number is done when it lands in the band.
Try it now
Then set number to 4500000 and press Use these numbers: the point moves left, so the dot jumps left, ÷ 10 at a time, and the power goes up.
Order of magnitude. The power of ten in scientific notation is the number's order of magnitude, its size class. 4.5 × 10⁶ is of order 6; 3.4 × 10⁻⁴ is of order −4. Two numbers of the same order are within ten times of each other. Each order up is ten times bigger.
Prefixes name the common orders:
prefix
power
example
nano (n)
10⁻⁹
a nanosecond
micro (µ)
10⁻⁶
a microsecond
milli (m)
10⁻³
a millimeter
kilo (k)
10³
a kilobyte
mega (M)
10⁶
a megabyte
giga (G)
10⁹
a gigabyte
tera (T)
10¹²
a terabyte
Multiplying and dividing. Multiply the front numbers, and add the powers, just as the laws of powers say. Dividing divides the fronts and takes the powers away:
(2×103)(4×105)=8×1083×1026×108=2×106
In plain words: the front numbers do ordinary arithmetic, and the powers of ten follow the laws.
If the front lands outside 1 to 10, shift it: (5 × 10³)(4 × 10⁴) = 20 × 10⁷, and 20 = 2 × 10¹, so the answer is 2 × 10⁸.
Rank by the power first: −4 is smallest, then −3, then −2. That's 0.0009, 0.007 and 0.02.
Converting back. To turn 3.4 × 10⁻⁴ into an ordinary number, move the point the other way: 4 places left, filling the gaps with zeros, to get 0.00034. For 4.5 × 10⁶, move it 6 places right: 4,500,000.
Two sense checks catch most slips. A negative power always gives a number less than 1. A positive power gives a number of 10 or more. And for a big number, the power counts the digits after the first one: 4,500,000 has 6 digits after the 4, so it's 4.5 × 10⁶.
In your own words
In your own words: why does comparing two numbers in scientific notation start with the powers of ten?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: e-notation
Purpose. Python writes scientific notation with an e, short for "times ten to the power". 3e-4 means 3 × 10⁻⁴, and 1.2e8 means 1.2 × 10⁸.
It runs right here in your browser, with nothing to install.
A number with an e is always a float. Python also prints very big and very small floats in e-notation on its own, so you need to read it even if you never type it:
Python writes a float out in full when it's at least 0.0001 and below 10¹⁶, and switches to e-notation outside that range: 0.00001 comes out as 1e-05, and 10¹⁶ as 1e+16. In 1e+16, the + just says the power is positive.
One number, three ways. Python's own choice is only one way to show a float. An f-string can ask for the style you want: :.1e always writes e-notation with one digit after the point, and :.6f always writes the number out to six places, like the :.2f from lesson 7. Step through a learning rate and one ten times smaller, each printed three ways. Watch what's printed so far: which style does plain print pick for each?
A named mistake: 10e-4 is not 10⁻⁴. The e already means "times ten to the power", so 10e-4 is 10 × 10⁻⁴, ten times too big. For 10⁻⁴, write 1e-4:
print(10e-4, 1e-4)
Reading 1e-16 as zero. In lesson 7, 0.1 + 0.2 wasn't exactly 0.3. Take one from the other and a tiny crumb is left:
print(0.1 + 0.2 - 0.3)
That's about 5.6 × 10⁻¹⁷: sixteen zeros after the point before the first real digit. When you expect 0 and see a number with e-16 or e-17, read it as "zero plus rounding", not as a real amount.
Orders of magnitude with log10.math.log10(x) answers "ten to what power gives x?" So it reads off the order of magnitude. math.floor rounds down to a whole number, giving the power in scientific notation:
import math
print(math.log10(1000))
print(math.log10(0.001))
print(math.log10(4500))
print(math.floor(math.log10(4500)))
print(math.floor(math.log10(0.00034)))
4,500 is between 10³ and 10⁴, so its log10 is between 3 and 4, and rounding down gives order 3. For 0.00034, log10 is about −3.5, and rounding down goes to −4, matching 3.4 × 10⁻⁴.
Worked example
Worked example: how long will it take?
A computer does 2 × 10⁹ simple steps a second. How long does a job of 3 × 10¹² steps take, in seconds and in minutes?
Dividing in scientific notation: fronts 3 ÷ 2 = 1.5, powers 12 − 9 = 3, so 1.5 × 10³ = 1,500 seconds. Python agrees, and dividing by 60 gives 25 minutes. Checking the order of magnitude first, 10¹² ÷ 10⁹ = 10³, tells you to expect about a thousand seconds before you run anything.
Quick check
What does this print?
print(4.2e5 * 2e-3)
8.4e-15
8.4e+2
840.0
840
Show the answer
840.0
Fronts 4.2 × 2 = 8.4; powers 5 + (−3) = 2. That's 8.4 × 10² = 840, and e-notation numbers are floats, so 840.0.
Quick check
Which line stores ten to the power minus eight?
eps = 1e-8
eps = 10e-8
eps = 10 ^ -8
eps = -1e8
Show the answer
eps = 1e-8
1e-8 means 1 × 10⁻⁸.
Drill
Read and write e-notation
Set rate to 3 × 10⁻⁴, params to 1.2 × 10⁸, steps to params times 50, and order to the order of magnitude of rate (use math.floor and math.log10). Each starter line has a slip.
import math
rate = 3e4
params = 12e8
steps = params + 50
order = math.log10(rate)
print(rate, params, steps, order)
In the lesson, your code is checked as you write it.
In your own words
When Python prints 5.551115123125783e-17 where you expected 0, what does it mean?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: the numbers engineers type
Purpose. ML code is full of e-notation. A learning rate of 3e-4. A model with 1.2e8 parameters. An epsilon of 1e-8, a tiny number added to the bottom of a division so it can never divide by zero.
Memory from parameters. Each parameter is usually stored as a 32-bit float, which takes 4 bytes. Lesson 12 shows what's inside those 4 bytes. For now, the count is enough:
bytes=parameters×4
In plain words: count the numbers the model keeps, and give each one 4 bytes.
A model with 1.2 × 10⁸ parameters needs about half a gigabyte, before training adds its own extra copies. And an epsilon keeps a division safe even when the bottom is 0:
bottom = 0print(1 / (bottom + 1e-8))
Quick check
What does this print?
params = 7e9print(params * 4 / 1e9)
0.28
28.0
28
2.8e+10
Show the answer
28.0
7 × 10⁹ × 4 = 2.8 × 10¹⁰ bytes, and dividing by 10⁹ gives 28 gigabytes. It's a float, so 28.0.
Drill
Memory for three models
At 4 bytes per parameter, set small, mid and big to the memory in gigabytes (10⁹ bytes) of models with 3.5e8, 7e9 and 7e10 parameters. Then set times to how many times more memory big needs than small.
small = 3.5e8 * 4
mid = 7e9 * 4 / 1e6
big = 7e10 / 1e9
times = big - small
print(small, mid, big, times)
In the lesson, your code is checked as you write it.
Learning rates by orders of magnitude. A learning rate sets how big each training step is. Too big and training jumps around; too small and it crawls. Engineers try rates an order of magnitude apart, such as 1e-2, 1e-3 and 1e-4, then narrow in. So it helps to see at a glance that 3e-4 is ten times 3e-5, and a hundred times 3e-6.
print(3e-4 / 3e-5)
print(3e-4 / 3e-6)
Both answers are a hair off 10 and 100, the same float rounding as before.
Quick check
Which learning rate is ten times 3e-5?
3e-4
3e-50
30e-4
3e-6
Show the answer
3e-4
One order of magnitude up: the power goes from −5 to −4.
The picture. On a number line where each power of ten is one equal step, a search like this is easy to see. Press Next to narrow in from five first tries.
In the real tools. In PyTorch, the library most models are built with, the learning rate goes to the optimizer, the part of the code that makes each training step. Here one is set up for a small model with lr=3e-4, and the last line reads the rate back from the optimizer's settings:
In your own words
Why do engineers try learning rates like 1e-2, 1e-3 and 1e-4 instead of 0.01, 0.02 and 0.03?
Write a sentence or two, then compare it with a good answer in the lesson.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write