Building a network runs its formulas forward: widths give parameters, parameters give memory, and the step gives the learning rate. Planning runs them backward. You start from what you have, a GPU's memory, a compute budget or a deadline, and solve for the width, the model size, the batch or the step. This lesson is transposing each formula of the network you just built, checking every plan with a round trip, and saying which way to round.
What you'll be able to do
Find a layer's width, or a hidden layer's width, from a parameter count, and round it the safe way
Turn memory, compute and data formulas around for the model size or the data a budget allows
Combine C ≈ 6ND with the rule of thumb D ≈ 20N to get N = √(C/120), and say that both are approximations
Free the head size d from the attention scale, and the bias b that makes a neuron hit a target activation
Solve a training plan for its batch size and its learning-rate schedule for a step, checking every plan with a round trip in code
Quick check
Have a guess before you start. A dense layer has 100 inputs and 5,050 parameters: a weight for every input and output pair, plus one bias per output. How many outputs?
Purpose. Plan a model from its budgets: turn each formula of a working network around for the size you need.
A dense layer gives every output a weight for each input, plus one bias, as in lesson 110. Its parameters, the numbers training sets, number P = n_in·n_out + n_out.
To find n_out, notice it appears twice. Factor it out, as in lesson 32: P = n_out(n_in + 1), so n_out = P/(n_in + 1). To find n_in, take the biases off first: n_in = (P − n_out)/n_out.
A network with d inputs, one hidden layer of width h and k outputs has P = dh + h + hk + k. Collect the h terms: P = h(d + k + 1) + k, so h = (P − k)/(d + k + 1).
Conditions. Widths are whole numbers. If the formula gives 41.5, no width has exactly that many parameters. For a budget, round down: 42 would go over it.
Quick check
From an earlier lesson: a network has 3 inputs, 4 hidden units and 3 outputs. How many parameters?
28
31
24
10
Show the answer
31
W₁ has 3 × 4 = 12, b₁ has 4, W₂ has 4 × 3 = 12 and b₂ has 3: 31. With the formula, h(d + k + 1) + k = 4 × 7 + 3 = 31.
Memory. Training with an Adam-style optimizer keeps about four numbers per parameter: the weight, its gradient, and the optimizer's two running averages. At b bytes a number, M ≈ 4Pb, so P ≈ M/(4b). At 4 bytes that's 16 bytes a parameter. Mixed precision (2-byte weights and gradients, plus a 4-byte master copy and averages) comes to about the same: 2 + 2 + 4 + 4 + 4 = 16. It's an approximation: the activations a batch makes need memory too.
Compute. Training a model with N parameters on D tokens, the word pieces a language model reads, takes about C ≈ 6ND FLOPs (single multiplies or adds): roughly 2 per parameter per token forward and 4 backward. So D = C/(6N), and N = C/(6D). It's an approximation too: it leaves out the attention scores' own work.
Combining. A rule of thumb from a 2022 study (the Chinchilla rule) says the compute-optimal split, the lowest loss for a fixed C, uses D ≈ 20N: twenty tokens per parameter. It's an approximation, not a law: many teams train smaller models on more tokens, since they're cheaper to run. Substitute it, as in lesson 39: C ≈ 6N·20N = 120N². N is squared, so freeing it takes a square root. Watch it with N in billions, so C comes out in units of 10¹⁸ FLOPs.
Try it now
Change N to 4 and press Use these numbers: twice the model needs 4 times the compute.
Symbols and formula. The two moves you'll use most:
P=nout(nin+1)⇒nout=nin+1P,C≈120N2⇒N≈120C
In plain words: a letter that appears twice gets factored out, then divided free. A letter that's squared gets a square root, and only the positive one makes sense for a size.
Named mistakes. Dividing P by n_in, which forgets the biases. Sizing a training run for the weights alone, which forgets the × 4. Halving the bytes for the weights alone, as if mixed precision halved the memory. Mixing units: here 1 GB is 10⁹ bytes, and C in units of 10¹⁸ FLOPs gives N in billions, since √10¹⁸ = 10⁹.
Quick check
A dense layer has 32 outputs and 3,232 parameters. How many inputs?
101
3,200
97.9
100
Show the answer
100
Take the 32 biases off: 3,232 − 32 = 3,200 weights. Each output has one per input: 3,200 ÷ 32 = 100. Check: 100 × 32 + 32 = 3,232.
Quick check
A 24 GB GPU (24 × 10⁹ bytes) trains with an Adam-style optimizer at 4 bytes a number. About how many parameters fit?
Under C ≈ 6ND and D ≈ 20N, why does 100 times the compute buy a model only 10 times bigger?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: a planning toolkit with round trips
Purpose. Write each rearrangement as a function, then run its answer forward through the original formula. If the round trip doesn't give the budget back, the rearrangement is wrong.
It runs right here in your browser, with nothing to install.
The first round trip gives 8,256 back, so the rearrangement is right.
A budget of 1,000 gives h = 41.54. math.floor rounds down to 41, which uses 987 parameters. The second check shows why: 42 would need 1,011, over budget.
Worked example
Worked example: the bias that hits a target activation
A sigmoid neuron gives a = σ(w·x + b), as in lesson 109. Peel the sigmoid with the logit from lesson 35: ln(a/(1 − a)) = w·x + b. Take w·x away: b = ln(a/(1 − a)) − w·x. The condition: a must be strictly between 0 and 1.
import math
import numpy as np
defsigmoid(z):
return1 / (1 + math.exp(-z))
defbias_for(a, w, x):
if a <= 0or a >= 1:
returnNonereturn math.log(a / (1 - a)) - float(np.dot(w, x))
w = np.array([0.5, -1.0])
x = np.array([3.0, 1.0])
b = bias_for(0.9, w, x)
print(round(b, 4), round(sigmoid(float(np.dot(w, x)) + b), 4), bias_for(1, w, x))
In the lesson you can change this and run it.
w·x = 1.5 − 1 = 0.5, and the logit of 0.9 is ln 9 ≈ 2.1972. So b ≈ 1.6972. The round trip puts it back into the neuron and gets 0.9. A target of exactly 1 has no answer: the sigmoid never reaches 1, however large the bias, so the function returns None.
The attention scale.lesson 113 divides each score by √d, where d is the length of the query and key vectors (the head size): s = q·k/√d. To free d, multiply by √d and divide by s: √d = q·k/s. Then square both sides: d = (q·k/s)². The conditions: s isn't 0, and s has the same sign as q·k, since √d is positive. A head size is a whole number, so a d like 15.7 means the scores were rounded.
Quick check
What does this print?
import math
defhead_size(raw, scaled):
return (raw / scaled) ** 2
d = head_size(8.0, 2.0)
print(d, 8.0 / math.sqrt(d))
4.0 4.0
16.0 2.0
2.0 2.0
16.0 0.5
Show the answer
16.0 2.0
√d = 8 ÷ 2 = 4, so d = 16.0. The round trip divides 8 by √16 = 4 and gives back the scaled score, 2.0.
Quick check
A run has C = 1.2 × 10²¹ FLOPs and 100 billion tokens of data. With C ≈ 6ND, about how many parameters N?
2 billion
3.16 billion
7.2 × 10³²
12 billion
Show the answer
2 billion
Here D is known, so one division frees N: N = C/(6D) = 1.2 × 10²¹ ÷ (6 × 10¹¹) = 2 × 10⁹. Check: 6 × 2 × 10⁹ × 10¹¹ = 1.2 × 10²¹.
Drill
Fix the planner
outputs_for forgets the biases and head_size takes a root where it needs a square. Fix both, so each round trip gives back what went in.
import math
defoutputs_for(P, n_in):
return P / n_in
defhead_size(raw, scaled):
return math.sqrt(raw / scaled)
print(outputs_for(8256, 128), head_size(8.0, 2.0))
In the lesson, your code is checked as you write it.
In your own words
Why run every planned number back through the original formula?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: sizing a training run
Purpose. Before a run starts, engineers fix its budgets: GPU memory, total compute and a deadline. Then they turn the formulas around for the model size, the data, the batch and the learning-rate schedule. Every training config and every scaling study is built this way.
A deadline. steps = epochs·n/batch for n examples, and the run takes T = steps × t at t seconds a step. Substitute: T = epochs·n·t/batch, so batch = epochs·n·t/T. This assumes a step takes the same time whatever the batch, which holds only while the GPU has room to spare, so measure t at the batch you pick. Round the batch up: a smaller batch means more steps, and the deadline is missed.
import math
defbatch_for_deadline(n, epochs, sec_per_step, T):
return math.ceil(epochs * n * sec_per_step / T)
defrun_time(n, epochs, batch, sec_per_step):
return math.ceil(epochs * n / batch) * sec_per_step
b = batch_for_deadline(100000, 3, 0.25, 3600)
print(b, run_time(100000, 3, b, 0.25), run_time(100000, 3, b - 1, 0.25))
The trail club's notes model has 100,000 examples, 3 epochs, 0.25 s a step and one hour. The formula gives 20.83, so the batch is 21: 3,571.5 s, inside 3,600. A batch of 20 would take 3,750 s.
Schedules. A learning-rate schedule sets the step size for each training step. Warmup grows the rate in a straight line, lr = lr_max·step/W, so step = lr·W/lr_max. Linear decay from step W to 0 at step S gives lr = lr_max·(S − step)/(S − W), so step = S − (lr/lr_max)(S − W). Exponential decay, lr = lr_max·γ^(step − W) with 0 < γ < 1, peels with a log: step = W + ln(lr/lr_max)/ln γ. Both logs are negative, so the step comes out after W. Cosine decay over T steps, with t counted from the start of the decay, gives lr = lr_max·(1 + cos(πt/T))/2. Peel it: t = T·arccos(2·lr/lr_max − 1)/π. Arccos returns the principal value, from 0 to π, which is exactly the range πt/T covers. Watch it with T = 1,000, showing lr as a share of lr_max.
Quick check
A rate warms up to lr_max by step W = 500, then decays linearly to 0 at step S = 5,500. At which step after warmup is it half of lr_max?
import math
defwarm_step(lr, lr_max, W):
return lr * W / lr_max
defdecay_step(lr, lr_max, W, gamma):
return W + math.log(lr / lr_max) / math.log(gamma)
print(warm_step(5e-4, 1e-3, 2000), math.ceil(decay_step(5e-4, 1e-3, 2000, 0.999)))
2000.0 2693
1000.0 693
1000.0 2693
1000 2692
Show the answer
1000.0 2693
Warmup reaches half the top rate at 0.5 × 2,000 = 1,000.0. The decay, lr_max·0.999^(step − 2000), halves after ln 0.5 ÷ ln 0.999 ≈ 692.8 more steps: 2,692.8, rounded up to 2693. A rate that rises and then falls hits most values twice.
Drill
Plan the run
Write batch_for_deadline(n, epochs, sec_per_step, T), the smallest whole batch that finishes in T seconds, and step_for_cosine(lr, lr_max, T), the first whole step at which cosine decay has fallen to lr.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
Learn to make any letter of any formula the subject: layers undone in reverse order, fractions, a letter that appears twice, powers, roots, quadratics, logs, inequalities, trigonometry, vectors, matrices, calculus, probability, statistics and the formulas inside AI models, each checked in Python.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write