Dropout

Why scale the survivors up?

Think of a bakery that usually has 4 bakers. One day, 1 of them calls in sick. If the other 3 work at their normal pace, the shop sells fewer loaves. To keep the shelves as full as usual, each baker who shows up has to work a bit harder.

Dropout faces the same problem. Switching some neurons off makes the total signal smaller. The next layer would then see weaker inputs in training than it sees at test time. So dropout boosts each survivor to make up the difference. Here $x$ is one activation before dropout:

In plain words: a neuron that survives gets divided by its chance of surviving, $1 - p$. If each neuron survives 3 times out of 4, each survivor is divided by 0.75. That makes it a third bigger.

Here's a small example. Take $p = 0.5$ and an activation of 3. Half the time it becomes 0. The other half it becomes 3 ÷ 0.5 = 6. Averaged over many steps, that's (0 + 6) ÷ 2 = 3, the same as the value you started with.

This trick is called inverted dropout. Because of the scaling, the average value of each activation is the same in training and in evaluation. So nothing needs adjusting at test time.

p = 0.25
scale = 1 / (1 - p)
x = [2.0, 4.0, 1.0, 3.0]
mask = [1, 0, 1, 1]          # the second neuron is switched off this step
out = [v * m * scale for v, m in zip(x, mask)]
print("scale:", round(scale, 3))
print("this step:", [round(v, 3) for v in out])
average = [(1 - p) * v * scale + p * 0 for v in x]
print("average over many steps:", [round(v, 3) for v in average])

Why it works

There are two ways to see it.

• No single neuron can be relied on. Any neuron might be missing on the next step. So the network spreads what it learns across many neurons. It builds backup features that work on new data. Without dropout, neurons can form fragile teams that only work together. Those teams tend to memorize the training set.
• A crowd of smaller networks. Each step switches off a different random set of neurons. So each step trains a slightly different smaller network, and they all share the same weights. At test time, the full network behaves roughly like the average of all of them. You met this idea in random forests. Many models voting together, called an ensemble, usually beat one model alone.

Seeing the effect

The next example uses scikit-learn's digits dataset. Each example is a small 8×8 image of a handwritten digit, so 64 pixel values. The task is to say which digit, 0 to 9, it shows.

The network has two hidden layers of 192 neurons each. That's far too big for only 150 training images. It reaches 100% training accuracy with or without dropout. The real test is the validation set: images the model never trained on. The code trains the network twice, once with $p = 0$ (no dropout) and once with $p = 0.5$.

A few names in the code: nn.Dropout(p) goes right after each ReLU. Adam is an optimizer, the rule that updates the weights; it gets its own lesson. The loss is cross-entropy, from Phase 2.

import torch
import torch.nn as nn
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

digits = load_digits()
X = torch.tensor(digits.data / 16.0, dtype=torch.float32)
y = torch.tensor(digits.target)
X_train, X_val, y_train, y_val = train_test_split(X, y, train_size=150, random_state=0, stratify=y)
loss_fn = nn.CrossEntropyLoss()

for p in [0.0, 0.5]:
    torch.manual_seed(0)
    model = nn.Sequential(nn.Linear(64, 192), nn.ReLU(), nn.Dropout(p),
                          nn.Linear(192, 192), nn.ReLU(), nn.Dropout(p),
                          nn.Linear(192, 10))
    opt = torch.optim.Adam(model.parameters(), lr=3e-3)
    for epoch in range(120):
        model.train()
        opt.zero_grad()
        loss_fn(model(X_train), y_train).backward()
        opt.step()
    model.eval()
    with torch.no_grad():
        train_acc = (model(X_train).argmax(1) == y_train).float().mean().item()
        val_acc = (model(X_val).argmax(1) == y_val).float().mean().item()
        val_loss = loss_fn(model(X_val), y_val).item()
    print(f"dropout {p}: train accuracy {train_acc:.3f}   val accuracy {val_acc:.3f}   val loss {val_loss:.3f}")

With dropout, the validation loss is much lower and the accuracy is a little higher.

Why does the loss change so much more than the accuracy? Cross-entropy loss punishes a wrong answer given with high confidence very hard. The model without dropout is overconfident: when it gets a digit wrong, it's very sure of its wrong answer. Dropout keeps its predictions more cautious. Its probabilities are closer to how often it's actually right, which is called being well calibrated.

Accuracy gains from dropout are often modest, as here. The more honest probabilities and steadier training matter too.

Using it well

• Where to put it: after the activation function in fully connected layers. In transformers, it goes on the attention weights. It also goes on each sub-layer's output, before that output is added back to the running total called the residual stream. You'll meet both in later lessons.
• How much: a rate of 0.1 to 0.5. Large models trained on huge datasets often use little or none, because they rarely see the same example twice. Fine-tuning on a small dataset is where dropout earns its keep.
• Always call model.eval() before you evaluate or generate. If you forget, dropout keeps switching neurons off, and predictions are randomly damaged. This is the most common dropout bug.
• Dropout makes training loss noisier and a bit higher. Judge it by validation results.

For marketers: it's like making sure a campaign doesn't depend on one channel or one star ad. Imagine you randomly remove pieces while planning. You end up with a plan that still works when any one piece underperforms.