Phase 4 · Deep learning · Training

Debugging training

Read loss curves and fix what's wrong.

About 35 minutes · A free account opens the whole lesson

Create a free account Try part of it here

Already have an account? Sign in to open this lesson

Why it matters

Bugs in a neural network rarely crash. The code runs, the loss prints, and the model is quietly wrong. The cause might be one missing line, labels in the wrong order, or a learning rate 100 times too big. Experienced practitioners still write these bugs. What sets them apart is a routine for finding them. This lesson gives you that routine. You'll use it again whenever a fine-tuning job or an eval "doesn't work".

What you'll be able to do

Recognize the loss-curve signatures of common training bugs
Run the standard sanity checks: initial loss, overfitting one batch, a baseline
Narrow a problem down systematically instead of changing many things at once
Diagnose a failing run from its numbers

What it covers

Bugs that don't crash
The first number: the starting loss
Reading the fingerprints
The routine
Checks that catch the classic slips
Worked example: is this starting loss right?
Beyond this network

Bugs that don't crash

When ordinary code has a bug, it usually stops with an error message. A training bug is different. The code keeps running and the loss keeps printing. Nothing tells you that something is wrong.

Think of a doctor. A patient doesn't arrive saying which illness they have. They arrive with symptoms: a fever, a cough, a rash. A good doctor knows which symptoms go with which illness. They also run a few standard tests before trying anything drastic.

Debugging training works the same way. The symptoms are the numbers a run prints. This lesson teaches you to read them, and gives you the standard tests.

Five runs, four bugs

The code below trains the same small network five times on the same data. The network looks at 8×8 pictures of handwritten digits and guesses which digit, 0 to 9, each one shows. One run is correct. Each of the other four has one classic bug planted in it:

no zero_grad: old gradients are never cleared, so they pile up.
learning rate 1.0: the learning rate is 100 times bigger than in the correct run, which uses 0.01.
labels shuffled: the right answers are mixed up, so they no longer match the pictures.
softmax before the loss: softmax gets applied twice, once by hand and once inside the loss.

For each run, the code prints the loss at the first and last step. It also prints the accuracy on 500 validation digits the model never trained on. Run it and compare the five lines:

import torch
import torch.nn as nn
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

digits = load_digits()
X = torch.tensor(digits.data / 16.0, dtype=torch.float32)
y = torch.tensor(digits.target)
X_train, X_val, y_train, y_val = train_test_split(X, y, train_size=500, test_size=500, random_state=0)
loss_fn = nn.CrossEntropyLoss()

def run(bug=None, steps=100):
    torch.manual_seed(0)
    model = nn.Sequential(nn.Linear(64, 64), nn.ReLU(), nn.Linear(64, 10))
    opt = torch.optim.Adam(model.parameters(), lr=1.0 if bug == "learning rate 1.0" else 1e-2)
    targets = y_train[torch.randperm(len(y_train))] if bug == "labels shuffled" else y_train
    losses = []
    for step in range(steps):
        if bug != "no zero_grad":
            opt.zero_grad()
        out = model(X_train)
        if bug == "softmax before the loss":
            out = out.softmax(dim=1)          # CrossEntropyLoss already applies softmax
        loss = loss_fn(out, targets)
        loss.backward()
        opt.step()
        losses.append(loss.item())
    with torch.no_grad():
        acc = (model(X_val).argmax(1) == y_val).float().mean().item()
    return losses, acc

for bug in [None, "no zero_grad", "learning rate 1.0", "labels shuffled", "softmax before the loss"]:
    losses, acc = run(bug)
    print(f"{str(bug or 'correct'):<24} loss {losses[0]:.2f} → {losses[-1]:.2f}   val accuracy {acc:.2f}")

It runs right here in your browser, with nothing to install.

(As in the rest of this phase, torch is the course's in-browser PyTorch subset.)

The first number: the starting loss

Look at the first loss in each line. Every run starts at about 2.30. That's no coincidence.

An untrained model knows nothing yet. So it spreads its guesses about evenly: roughly a 1 in 10 chance for each digit. Cross-entropy loss, from Phase 2, scores a guess by how much probability it gave the right answer. A 1-in-10 guess always scores about 2.30.

The general rule needs two symbols. Let $C$ be the number of classes, the possible answers. Here $C = 10$ . The symbol $\ln$ is the natural logarithm, which Python writes as math.log.

\text{starting loss} \approx \ln C

In plain words: an untrained classifier should start with a loss close to the natural log of the number of classes.

Try it with small numbers. A model with 4 classes guesses 1 in 4, or 0.25, for each one. Its loss is minus the log of 0.25, which is about 1.39. That's the same as the log of 4. With 10 classes, each guess is 0.1, and minus the log of 0.1 is about 2.30.

Here's the same calculation in Python:

import math

for classes in [2, 4, 10]:
    p_right = 1 / classes          # even guessing
    print(classes, round(-math.log(p_right), 3))

It prints 0.693 for 2 classes, 1.386 for 4 and 2.303 for 10.

Why check this number? If the first loss is far above $\ln C$ , the model is confidently wrong before it has learned anything. That usually points to badly scaled inputs or bad starting weights.

Go deeper Why even guessing gives ln C

For one example, cross-entropy loss is minus the log of the probability given to the right class. Call that probability $p$ .

\text{loss} = -\ln p = -\ln \frac{1}{C} = \ln C

Read it as: with even guessing, $p$ is one over the number of classes. Minus the log of one over a number is the log of the number itself.

For 4 classes: $p = 0.25$ , and $-\ln 0.25 = \ln 4 \approx 1.39$ .

Quick check

Your untrained model sorts photos into 4 classes. What does this print, and so what starting loss should you expect?

import math
print(round(math.log(4), 2))

1.39
4.0
2.3
0.25

Show the answer

1.39

ln 4 is about 1.39. An untrained 4-class model guesses about 1 in 4 for each class, so its loss should start near 1.39.

Reading the fingerprints

After the first step, each bug leaves its own fingerprint on the numbers. This table matches what you see to its likely cause. Use it to match each of the four bugged runs above to a row.

What you see	Likely cause
training loss falls, but validation accuracy stays at chance	the labels don't match the inputs (shuffled or misaligned)
loss stuck near its starting value	learning rate far too high (dead units) or far too low; a frozen model; a detached graph
loss jumps around or rises	gradients piling up (no `zero_grad`), learning rate too high, bad data
loss stalls at a floor well above 0, accuracy decent	softmax applied twice (softmax before `CrossEntropyLoss`), a capped output, the wrong loss for the task
loss is `nan`	learning rate too high, log of 0, division by 0, exploding gradients

Some of those words need a short explanation:

Chance is the accuracy you'd get by guessing at random. With 10 digits, that's 1 in 10, or 10%.
Dead units are ReLU neurons that output 0 for every input. They get no gradient, so they stop learning. Huge steps can knock many of them into this state at once.
A frozen model has its weights switched off from updating.
A detached graph has a broken link between the loss and the weights. The backward pass can't reach the weights, so they never change.
nan means "not a number". The values grew so large, or so broken, that they stopped being numbers at all.

Here's what three of the bugs look like from the inside.

Shuffled labels. Picture flashcards with the wrong answers written on the back. A student can memorize every card, so the training loss falls. But nothing they learned helps with a new card. That's why validation accuracy stays at chance.

Quick check

Training loss falls but validation accuracy stays at chance. What's the likely cause?

The model is too small
The labels don't match the inputs
The GPU is too slow
The learning rate is too low

Show the answer

The labels don't match the inputs

A model can memorize wrong answers, but nothing it memorizes helps on new data.

No zero_grad. PyTorch adds each new gradient on top of the one already stored. It's like a shopping receipt that never gets cleared: each round adds to the total. By step 10, the update uses the sum of ten steps' gradients, most of them out of date. The model keeps pushing in old directions and overshoots, so the loss jumps around.

Softmax twice. CrossEntropyLoss expects raw scores and applies softmax itself. Applying it by hand first squashes every score to between 0 and 1. The best digit still has the highest score, so accuracy stays decent. But the loss can't get anywhere near 0.

Go deeper Why softmax twice puts a floor under the loss

After the first softmax, every score is between 0 and 1. So the scores going into the second softmax differ by at most 1. Even a perfect model can't give the right answer much probability.

Take the best case for 10 classes. The right digit scores 1, and the other nine score 0. Softmax raises the number $e$ (about 2.718) to the power of each score. Since $e^0 = 1$ , the nine wrong digits add up to 9.

p_{\text{right}} = \frac{e^1}{e^1 + 9}

Read it as: the right digit's share is $e$ divided by $e$ plus nine.

That's 2.718 divided by 11.718, about 0.23. The loss is minus the log of 0.23, about 1.46. So this run's loss can never fall below about 1.46, even when every answer is right.

import math
best = math.e / (math.e + 9)
print(round(best, 3), round(-math.log(best), 2))

It prints 0.232 and 1.46.

Try it now

Most training bugs are silent, so learn their fingerprints.

Quick check

Training loss steadily falls but validation accuracy stays at 10% for a 10-class problem. What's the most likely bug?

The training labels don't correspond to the inputs (shuffled or misaligned)
Dropout is too high
The learning rate is too low
The model is too small

Show the answer

The training labels don't correspond to the inputs (shuffled or misaligned)

The model is memorizing noise. It can fit the training pairs, like flashcards with wrong answers, but it learns nothing that carries over to new examples.

The routine

When a run looks wrong, work through these steps in order.

Check the starting loss. For a classifier, it should be about $\ln C$ , the log of the number of classes. Much higher means the outputs are confidently wrong from the start. Look at how the inputs are scaled and how the weights were set up.
Overfit one small batch. Take 10–20 examples and train on only those for a few hundred steps. A working model and loop should drive the loss almost to 0. It's like checking that a student can memorize ten flashcards before handing over the whole deck. If it can't, the bug is in the model, the loss or the loop. More data or more regularization won't fix it.
Compare with a baseline. A baseline is a simple method to beat. Examples: always guessing the most common class, a logistic regression, or last season's model. A network that can't beat a baseline is suspect, not impressive.
Look at the data. Print a few inputs next to their labels. Do this after every preprocessing step. Shuffled or misaligned labels are far more common than strange model bugs.
Watch the gradients. Per-layer gradient norms, from the previous lesson, measure how big each layer's gradients are. They show layers that are dead or exploding.
Change one thing at a time. Keep the random seed fixed while you debug. Then a difference between two runs comes from your change, not from luck.

Quick check

You train on 16 examples for 300 steps, and the loss stays high. What's the best next step?

Look for a bug in the model, the loss or the training loop
Add regularization
Collect more training data
Train on the full dataset instead

Show the answer

Look for a bug in the model, the loss or the training loop

A working setup should memorize 16 examples easily. If it can't, something in the code is broken, and more data or regularization won't help.

Checks that catch the classic slips

Call optimizer.zero_grad() once per step, before backward().
Use model.train() while training. Use model.eval() with torch.no_grad() while evaluating.
CrossEntropyLoss takes raw logits (the model's raw scores) and whole-number class labels. Don't apply softmax first. Don't one-hot the labels, which means turning each label into a list of 0s with a single 1.
Scale the inputs sensibly, and use the same scaling at test time. The digit pixels here run from 0 to 16, so the code divides them by 16.
Print the shapes at every stage of the first batch.
Give the optimizer model.parameters() of the model you're training, not of a copy.

Worked example

Worked example: is this starting loss right?

You build a 3-class sentiment model: positive, neutral or negative. Before training, run A prints a first loss of 1.12 and run B prints 5.70. Compare each with the natural log of 3.

import math

classes = 3
expected = math.log(classes)        # even guessing: ln 3
first_losses = {"run A": 1.12, "run B": 5.70}
for name, loss in first_losses.items():
    ratio = loss / expected
    print(f"{name}: starts at {loss}, expected {expected:.2f}, ratio {ratio:.1f}")

In the lesson you can change this and run it.

The log of 3 is about 1.10. Run A starts right next to it (ratio 1.0), so its setup looks healthy. Run B starts more than five times higher (ratio 5.2). Before learning anything, it is already confidently wrong. Check its input scaling and weight setup before training any further. (Python prints 5.70 as 5.7.)

Beyond this network

The same habits work on much bigger systems. Say a fine-tune "doesn't improve". Check its starting loss, and test whether it can overfit a single batch. Say an LLM feature "doesn't work". Compare it with a baseline, and read its actual inputs and outputs. The evals lessons in Phase 1 apply this same routine to prompts.

Quick check

A spam filter scores 95% accuracy on new emails. 95% of those emails are not spam. What should you conclude?

It's an excellent model
It may be no better than always answering "not spam"
The learning rate is too low
It's overfitting

Show the answer

It may be no better than always answering "not spam"

Always answering "not spam" also scores 95%. That's the most-common-class baseline, and this model hasn't beaten it yet.

Practice what you just read

4 code examples you can edit and run
a practice exercise that checks your code
a 5-question quiz
4 review cards for spaced repetition

Create a free account

Try a quiz question

What's the purpose of trying to overfit a single small batch?

To confirm the model, loss and training loop can learn at all before worrying about generalization
To get the best final model, since a model that masters one batch does well everywhere
To pick the batch size, by finding the smallest one that the model is able to memorize
To measure validation accuracy, without having to wait for a full pass over all the data

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Optimizers