Scaling laws

The Chinchilla formula

In 2022, a team at DeepMind (Hoffmann and colleagues) wrote one formula for loss that covers both model size and data. Their best model was called Chinchilla, and the formula usually goes by that name too:

Take it one piece at a time.

• $L(N, D)$ is the loss for a model with $N$ parameters trained on $D$ tokens.
• $E$ is the irreducible loss: the floor. Some text is unpredictable however good the reader is, so no model can get below it.
• $\frac{A}{N^{\alpha}}$ is the size penalty: extra loss because the model is too small. $A$ is a number fitted from the experiments. The exponent $\alpha$ (the Greek letter alpha) sets how fast the penalty shrinks as the model grows.
• $\frac{B}{D^{\beta}}$ is the data penalty: extra loss because the model saw too little text. $B$ is another fitted number. The exponent $\beta$ (beta) sets how fast this penalty shrinks as the data grows.

In plain words: loss = the floor + a penalty for too few parameters + a penalty for too little data. Each penalty is a power law. It shrinks as you add more, but never quite reaches zero.

Here's a worked example with made-up, easy numbers. Say $E = 2$, $A = 10$ and $B = 20$, and both exponents are 0.5. Take a model with 100 parameters trained on 400 tokens.

• Size penalty: 10 ÷ √100 = 10 ÷ 10 = 1.
• Data penalty: 20 ÷ √400 = 20 ÷ 20 = 1.
• Loss: 2 + 1 + 1 = 4.

Now make the model 100 times bigger, with 10,000 parameters. The size penalty drops to 10 ÷ 100 = 0.1. The data penalty stays at 1, so the loss is 3.1. Most of what's left above the floor is now the data penalty. Adding more parameters can't fix that.

E = 2
A, alpha = 10, 0.5
B, beta = 20, 0.5
D = 400
data_penalty = B / D**beta
for N in [100, 10_000]:
    size_penalty = A / N**alpha
    print(N, size_penalty, data_penalty, E + size_penalty + data_penalty)

The real numbers

The values below come from a 2024 re-check of the Chinchilla analysis by Epoch AI. The loss is in nats per token (the unit from the cross-entropy lesson), measured on the paper's own dataset. So compare these numbers with each other, not with other models.

import matplotlib.pyplot as plt
import numpy as np

E, A, B, alpha, beta = 1.82, 482.01, 2085.43, 0.3478, 0.3658

def loss(N, D):
    return E + A / N**alpha + B / D**beta

print("model size (data fixed at 1.4 trillion tokens)")
for N in [1e8, 1e9, 1e10, 1e11, 1e12]:
    print(f"  {N:7.0e} parameters   loss {loss(N, 1.4e12):.3f}")
print("training data (model fixed at 70 billion parameters)")
for D in [1e10, 1e11, 1e12, 1e13, 1e14]:
    print(f"  {D:7.0e} tokens       loss {loss(70e9, D):.3f}")

N = np.logspace(8, 12, 50)
plt.loglog(N, A / N**alpha, label="size penalty A / N^α")
plt.loglog(N, loss(N, 1.4e12) - E, label="loss above the floor (D = 1.4T)")
plt.xlabel("parameters")
plt.ylabel("loss above E")
plt.legend()

In the printout, 1e+09 means 1 followed by 9 zeros: a billion. Read the first table from the top. Each 10× in model size helps less than the one before. The loss drops by 0.44, then 0.20, then 0.09, then 0.04. Each 10× cuts the size penalty to about 45% of what it was.

On the chart, the size penalty alone is a straight line: a power law. The total loss above the floor bends flat on the right. There, the fixed data penalty is most of what's left. The numbers teach two things. Each 10× of size or data buys a smaller gain than the last. And growing only one of them stalls.

Spending a compute budget

Compute is the real limit. It costs money and time on GPUs, the chips that train models. From the last lesson, training compute is roughly:

In plain words: training compute is about 6 × parameters × tokens. Each parameter does about 6 calculations for every token the model trains on.

For example, a tiny model with 1,000 parameters trained on 50 tokens needs about 6 × 1,000 × 50 = 300,000 FLOPs. The worked example further down does the same sum for a real model.

So with a fixed budget, a bigger model means fewer tokens. It's like a fixed tank of fuel for a road trip: a bigger car goes fewer miles. Which split of the budget gives the lowest loss?

The code below tries 6,001 model sizes, from 10 million to 10 trillion parameters. It gives each one the tokens the budget can pay for, then keeps the size with the lowest loss.

import numpy as np

E, A, B, alpha, beta = 1.82, 482.01, 2085.43, 0.3478, 0.3658

def loss(N, D):
    return E + A / N**alpha + B / D**beta

def compute_optimal(C):
    N = np.logspace(7, 13, 6001)          # candidate model sizes
    D = C / (6 * N)                        # the tokens each one can afford
    best = loss(N, D).argmin()
    return N[best], D[best]

for C in [1e21, 1e23, 1e25]:
    N, D = compute_optimal(C)
    print(f"budget {C:.0e} FLOPs: {N:.1e} parameters, {D:.1e} tokens ({D / N:.0f} tokens per parameter)")

print("\nSame budget, two choices:")
print(f"  Gopher:     280B parameters, 300B tokens   loss {loss(280e9, 300e9):.3f}")
print(f"  Chinchilla:  70B parameters, 1.4T tokens   loss {loss(70e9, 1.4e12):.3f}")

The answer: grow size and data together, at roughly 20 tokens per parameter. The best split is called compute-optimal.

Before Chinchilla, labs had been making models bigger without adding much data. DeepMind's own Gopher had 280 billion parameters and read 300 billion tokens. It was badly undertrained. Chinchilla was a quarter of the size and read 4.7× as much data, for about the same compute. It beat Gopher across the board, and it was cheaper to run.

import math

C = 6 * 280e9 * 300e9        # Gopher's budget: 6 × parameters × tokens
N = math.sqrt(C / 120)       # the size that gets 20 tokens per parameter
D = 20 * N                   # tokens
print(f"budget {C:.1e} FLOPs")
print(f"{N:.1e} parameters, {D:.1e} tokens")
print(f"check: 6 × N × D = {6 * N * D:.1e}")

Predicting a big run from small ones

A baker trying a new cake for a wedding bakes a few small test cakes first. Labs do the same with scaling laws. They train a series of small models and fit the curve to the results. Then they predict how a run 1,000 times larger will turn out, before paying for it. OpenAI's GPT-4 technical report says its final loss was predicted from runs that used up to 10,000 times less compute.

Here is the idea with six pretend small runs. The code makes up their final losses from a hidden law, plus a little random noise. The law has the same shape as before: a floor plus a power-law penalty, now in terms of compute $C$. (Dividing by $10^{18}$ keeps the numbers a handy size.) Then curve_fit, from the SciPy library, finds the floor, the size and the exponent that best match the six losses.

import numpy as np
from scipy.optimize import curve_fit

def law(C, E, a, b):
    return E + a * (C / 1e18) ** (-b)

# pretend these are the final losses of six small training runs (a hidden law plus noise)
rng = np.random.default_rng(0)
small = np.logspace(17, 19, 6)
measured = law(small, 1.9, 1.2, 0.15) * (1 + rng.normal(0, 0.005, len(small)))

(E, a, b), _ = curve_fit(law, small, measured, p0=[2, 1, 0.1])
print(f"fitted law: loss = {E:.2f} + {a:.2f} × (C / 10^18)^-{b:.3f}")
for C in [1e19, 1e21, 1e23]:
    print(f"compute {C:.0e}: predicted {law(C, E, a, b):.3f}   true {law(C, 1.9, 1.2, 0.15):.3f}")

The fit only saw runs up to $10^{19}$ FLOPs. Even so, it predicts the loss at $10^{23}$, 10,000 times further out, to within 0.01. Real predictions are noisier than this, but the method is the same.

Beyond compute-optimal

"Compute-optimal" gives the lowest loss for a given training budget. But a model is trained once and then used billions of times. A smaller model is cheaper for every one of those requests. So labs now overtrain small models, far past 20 tokens per parameter. Meta trained Llama 3's 8-billion-parameter model on about 15 trillion tokens. That's nearly 2,000 tokens per parameter. Its loss kept improving long past the Chinchilla point, only more slowly.

Some caveats worth knowing:

• Loss isn't ability. Scaling laws predict loss. But specific skills can seem to switch on suddenly at some size. These are called "emergent abilities". Researchers still debate how much of that suddenness comes from scoring. Many benchmarks mark an answer fully right or fully wrong, with no partial credit.
• Data may run out. Some estimates suggest high-quality public text could be largely used up within a few years at current growth. That's why labs invest in synthetic data, better filtering and other sources.
• Other things scale too. Reasoning models (Phase 1's thinking effort) show that spending more compute at answer time also improves results predictably. Thinking longer is a second way to scale, on top of training.

For marketers: when a vendor's small model matches last year's large one, overtraining is often the reason. It's the same design, trained on far more data per parameter, at a much lower price per call.

print(round(15e12 / 8e9))