Index Ai Academy
Foundation · Foundation mathTransposition of formulas

Planning a model: rearranging the formulas of a working network

About 40 minLooking ahead: some prerequisites aren't done yet

Why this matters

Building a network runs its formulas forward: widths give parameters, parameters give memory, and the step gives the learning rate. Planning runs them backward. You start from what you have, a GPU's memory, a compute budget or a deadline, and solve for the width, the model size, the batch or the step. This lesson is transposing each formula of the network you just built, checking every plan with a round trip, and saying which way to round.

  • Find a layer's width, or a hidden layer's width, from a parameter count, and round it the safe way
  • Turn memory, compute and data formulas around for the model size or the data a budget allows
  • Combine C ≈ 6ND with the rule of thumb D ≈ 20N to get N = √(C/120), and say that both are approximations
  • Free the head size d from the attention scale, and the bias b that makes a neuron hit a target activation
  • Solve a training plan for its batch size and its learning-rate schedule for a step, checking every plan with a round trip in code
Quick check
Have a guess before you start. A dense layer has 100 inputs and 5,050 parameters: a weight for every input and output pair, plus one bias per output. How many outputs?

The math: one network, many unknowns

Purpose. Plan a model from its budgets: turn each formula of a working network around for the size you need.

A dense layer gives every output a weight for each input, plus one bias, as in lesson 110. Its parameters, the numbers training sets, number P = n_in·n_out + n_out.

To find n_out, notice it appears twice. Factor it out, as in lesson 32: P = n_out(n_in + 1), so n_out = P/(n_in + 1). To find n_in, take the biases off first: n_in = (P − n_out)/n_out.

A network with d inputs, one hidden layer of width h and k outputs has P = dh + h + hk + k. Collect the h terms: P = h(d + k + 1) + k, so h = (P − k)/(d + k + 1).

Conditions. Widths are whole numbers. If the formula gives 41.5, no width has exactly that many parameters. For a budget, round down: 42 would go over it.