Planning a model: rearranging the formulas of a working network
Why this matters
Building a network runs its formulas forward: widths give parameters, parameters give memory, and the step gives the learning rate. Planning runs them backward. You start from what you have, a GPU's memory, a compute budget or a deadline, and solve for the width, the model size, the batch or the step. This lesson is transposing each formula of the network you just built, checking every plan with a round trip, and saying which way to round.
- Find a layer's width, or a hidden layer's width, from a parameter count, and round it the safe way
- Turn memory, compute and data formulas around for the model size or the data a budget allows
- Combine C ≈ 6ND with the rule of thumb D ≈ 20N to get N = √(C/120), and say that both are approximations
- Free the head size d from the attention scale, and the bias b that makes a neuron hit a target activation
- Solve a training plan for its batch size and its learning-rate schedule for a step, checking every plan with a round trip in code
The math: one network, many unknowns
Purpose. Plan a model from its budgets: turn each formula of a working network around for the size you need.
A dense layer gives every output a weight for each input, plus one bias, as in lesson 110. Its parameters, the numbers training sets, number P = n_in·n_out + n_out.
To find n_out, notice it appears twice. Factor it out, as in lesson 32: P = n_out(n_in + 1), so n_out = P/(n_in + 1). To find n_in, take the biases off first: n_in = (P − n_out)/n_out.
A network with d inputs, one hidden layer of width h and k outputs has P = dh + h + hk + k. Collect the h terms: P = h(d + k + 1) + k, so h = (P − k)/(d + k + 1).
Conditions. Widths are whole numbers. If the formula gives 41.5, no width has exactly that many parameters. For a budget, round down: 42 would go over it.