What it means when training diverges

Diverge

Also written: diverges, diverged, divergence

When training blows up: the loss grows instead of settling, often because the learning rate is too big.

What it means

In machine learning, training diverges when the loss grows instead of settling, often until it hits infinity or NaN. The usual cause is a learning rate that's too big: each step overshoots the best value by more than the last one did, so the model gets further away with every update.

An example in Python

def train(lr, steps=6):
    w = 10.0
    losses = []
    for _ in range(steps):
        w -= lr * 2 * (w - 3)       # gradient descent on (w - 3) squared
        losses.append((w - 3) ** 2)
    return [round(x, 1) for x in losses]

print("lr 0.1:", train(0.1))
print("lr 1.1:", train(1.1))

What it prints

lr 0.1: [31.4, 20.1, 12.8, 8.2, 5.3, 3.4]
lr 1.1: [70.6, 101.6, 146.3, 210.7, 303.4, 436.9]

At a rate of 0.1 the loss falls steadily. At 1.1, nothing else changes, and the loss grows by the same factor on every step.

What to do about it

Lower the learning rate first, and a learning rate sweep will show you where the edge is. If that doesn't fix it, look at the data for bad values, unscaled columns or labels on the wrong scale. In deep networks, exploding gradients can do the same thing, and gradient clipping is the usual fix. Check the first few steps: a loss that's already climbing won't recover.

Try it now

Run 10 steps at 0.02 (too small). Then reset and try 0.4, 0.95 and 1.05. Watch the status line.

Other terms from the same lesson

cosine decay: A schedule that lowers the learning rate along a smooth half-wave curve, slowly at first, then faster, then gently down to near zero.

learning rate: How big a step training takes each time. Too big overshoots; too small crawls.

learning-rate schedule: A plan for changing the learning rate as training goes on, usually starting larger and ending smaller.

learning-rate sweep: Training briefly with several learning rates, often each about three times the last, and keeping the one whose loss falls best.

warmup: Starting training with a tiny learning rate and raising it over the first steps, so early, wild updates don't knock the model off course.

Never trained an AI?

Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.

Teach it to write