Capstone: train a classifier

Step 2: checks before you trust a run

The Debugging training lesson gave you a routine for runs that go wrong. Use two of its checks here, before any real training.

Check 1: the starting loss. An untrained network knows nothing. With two classes, churn or stay, it should give each one about a 50% chance. Cross-entropy loss scores a guess as minus the log of the probability it gave the right answer. So the first loss should be close to $\ln 2$. Here $\ln$ is the natural logarithm, which Python writes as math.log.

In plain words: an untrained two-class model guesses 50/50, so its loss should start near the log of 2, which is about 0.693.

Try it with small numbers. The right answer gets a probability of 0.5. Minus the log of 0.5 is 0.693. That's the same as the log of 2. In Python:

import math
p_right = 0.5                     # a 50/50 guess
print(round(-math.log(p_right), 3), round(math.log(2), 3))

It prints 0.693 twice. The big cell below prints math.log(2) next to the network's real starting loss, so you can compare them.

Check 2: overfit a tiny batch. Train on only 16 customers for 300 steps. A working model and training loop should memorize them and drive the loss close to 0. If it can't, the bug is in the model, the loss or the loop. More data won't fix it.

The baseline. A baseline is a simple model the network has to beat. Here it's logistic regression from Phase 3. Its setting C=0.1 means fairly strong regularization: it keeps the weights small. You score it with AUC on the validation set. AUC, from Phase 3, is the chance that the model ranks a random churner above a random customer who stayed. A score of 0.5 is a coin flip, and 1.0 is perfect.

import math
import numpy as np
import pandas as pd
import requests
import torch
import torch.nn as nn
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split

df = pd.DataFrame(requests.get("https://api.gradient.test/v1/customers").json()["data"])
numeric = ["tenure_months", "orders_last_year", "avg_order_value", "days_since_last_order",
           "discount_share", "email_open_rate", "support_tickets", "trail_club"]
categorical = ["region", "channel", "top_category"]
df["email_missing"] = df["email_open_rate"].isna().astype(float)
dev, test = train_test_split(df, test_size=0.2, random_state=7, stratify=df["churned"])
train, val = train_test_split(dev, test_size=0.25, random_state=7, stratify=dev["churned"])

def prepare(fit, *frames):
    """Learn medians, means, scales and categories from `fit` only, then apply them to every frame."""
    med = fit[numeric].median()
    mu, sd = fit[numeric].fillna(med).mean(), fit[numeric].fillna(med).std()
    cats = pd.get_dummies(fit[categorical].replace("", "unknown")).columns
    out = []
    for f in frames:
        x = (f[numeric].fillna(med) - mu) / sd
        d = pd.get_dummies(f[categorical].replace("", "unknown")).reindex(columns=cats, fill_value=0)
        out.append(torch.tensor(np.c_[x.to_numpy(float), f[["email_missing"]].to_numpy(float), d.to_numpy(float)], dtype=torch.float32))
    return out

X_train, X_val, X_test = prepare(train, train, val, test)
y_train, y_val, y_test = (torch.tensor(f["churned"].to_numpy()) for f in (train, val, test))
print("shapes:", tuple(X_train.shape), tuple(X_val.shape), tuple(X_test.shape), f"  churn rate {y_train.float().mean().item():.1%}")

torch.manual_seed(0)
model = nn.Sequential(nn.Linear(X_train.shape[1], 32), nn.ReLU(), nn.Linear(32, 2))
with torch.no_grad():
    start = nn.functional.cross_entropy(model(X_train), y_train).item()
print(f"starting loss {start:.3f}   (two classes: ln 2 = {math.log(2):.3f})")

opt = torch.optim.Adam(model.parameters(), lr=0.01)       # can it memorize 16 customers?
for _ in range(300):
    opt.zero_grad()
    loss = nn.functional.cross_entropy(model(X_train[:16]), y_train[:16])
    loss.backward()
    opt.step()
print(f"loss after overfitting 16 examples: {loss.item():.4f}")

baseline = LogisticRegression(C=0.1, max_iter=2000).fit(X_train.numpy(), y_train.numpy())
print(f"logistic regression validation AUC: {roc_auc_score(y_val.numpy(), baseline.predict_proba(X_val.numpy())[:, 1]):.3f}")

Here's how to read the output:

• The first line shows the three sets: 360, 120 and 120 rows. All three have the same number of columns, because prepare lined them up. The line also shows the churn rate in the training set.
• The starting loss sits near 0.693. A much higher number would mean the network is confidently wrong before it learns anything.
• The loss after 300 steps on 16 customers is close to 0. The network can memorize, so the plumbing works.
• The logistic regression's validation AUC is the bar. The network has to clear it to earn its place.

import math
print(round(math.log(10), 1))

Step 3: train with the brakes on

A network has many weights. It can memorize 360 customers instead of learning the pattern behind them. The final model uses three brakes against that. Together they're called regularization.

• Dropout (0.2) switches off about 20% of the hidden units, picked at random, at each training step. The network can't rely on any single unit.
• AdamW is the Adam optimizer plus weight decay. At every step it shrinks each weight a little toward 0 (weight_decay=0.01). A large weight has to keep earning its size.
• Early stopping watches the validation AUC and keeps the best version of the model.

The network has two hidden layers of 32 units. Each layer is followed by ReLU and dropout. Training uses mini-batches of 32 customers, in a fresh random order each time. One full pass through the training set is called an epoch.

Early stopping

Think of baking bread. You check the loaf every few minutes. It gets better, and then it starts to burn. You want the loaf from its best moment, not the last one.

Early stopping does the same with the model. After every epoch:

1. Measure the AUC on the validation set.
2. If it beats the best so far, save a copy of all the weights. PyTorch calls this copy the state_dict.
3. If 20 epochs pass with no new best, stop. That waiting period is called the patience.

At the end, load the saved best weights back into the model.

Two small details matter when you measure. model.eval() switches dropout off, so every unit takes part. torch.no_grad() tells PyTorch not to track gradients, because nothing is learned while you score. The helper churn_prob in the code does both. It returns each customer's softmax probability for class 1, churn.

val_auc = [0.70, 0.74, 0.77, 0.78, 0.78, 0.77, 0.79, 0.78, 0.78, 0.77, 0.76]
patience = 3
best_auc, best_epoch = -1.0, 0
for epoch, auc in enumerate(val_auc):
    if auc > best_auc:
        best_auc, best_epoch = auc, epoch
    if epoch - best_epoch >= patience:
        break
print(f"best AUC {best_auc} at epoch {best_epoch}, stopped after epoch {epoch}")

Step 4: run it, then score each model once

The code below repeats the data preparation from the first big cell, so it runs on its own. Then it trains the network with early stopping. Last, it scores both models on the test set, once each.

import numpy as np
import pandas as pd
import requests
import torch
import torch.nn as nn
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split

df = pd.DataFrame(requests.get("https://api.gradient.test/v1/customers").json()["data"])
numeric = ["tenure_months", "orders_last_year", "avg_order_value", "days_since_last_order",
           "discount_share", "email_open_rate", "support_tickets", "trail_club"]
categorical = ["region", "channel", "top_category"]
df["email_missing"] = df["email_open_rate"].isna().astype(float)
dev, test = train_test_split(df, test_size=0.2, random_state=7, stratify=df["churned"])
train, val = train_test_split(dev, test_size=0.25, random_state=7, stratify=dev["churned"])

def prepare(fit, *frames):
    med = fit[numeric].median()
    mu, sd = fit[numeric].fillna(med).mean(), fit[numeric].fillna(med).std()
    cats = pd.get_dummies(fit[categorical].replace("", "unknown")).columns
    out = []
    for f in frames:
        x = (f[numeric].fillna(med) - mu) / sd
        d = pd.get_dummies(f[categorical].replace("", "unknown")).reindex(columns=cats, fill_value=0)
        out.append(torch.tensor(np.c_[x.to_numpy(float), f[["email_missing"]].to_numpy(float), d.to_numpy(float)], dtype=torch.float32))
    return out

X_train, X_val, X_test = prepare(train, train, val, test)
y_train, y_val, y_test = (torch.tensor(f["churned"].to_numpy()) for f in (train, val, test))

def churn_prob(model, X):
    model.eval()
    with torch.no_grad():
        return torch.softmax(model(X), dim=1)[:, 1].numpy()

torch.manual_seed(0)
model = nn.Sequential(nn.Linear(X_train.shape[1], 32), nn.ReLU(), nn.Dropout(0.2),
                      nn.Linear(32, 32), nn.ReLU(), nn.Dropout(0.2), nn.Linear(32, 2))
opt = torch.optim.AdamW(model.parameters(), lr=3e-3, weight_decay=0.01)
best_auc, best_epoch, best_state = -1.0, 0, None
for epoch in range(150):
    model.train()
    order = torch.randperm(len(X_train))
    for i in range(0, len(X_train), 32):
        idx = order[i:i + 32]
        opt.zero_grad()
        nn.functional.cross_entropy(model(X_train[idx]), y_train[idx]).backward()
        opt.step()
    auc = roc_auc_score(y_val.numpy(), churn_prob(model, X_val))
    if auc > best_auc:
        best_auc, best_epoch = auc, epoch
        best_state = {k: v.clone() for k, v in model.state_dict().items()}
    if epoch - best_epoch >= 20:                    # no improvement for 20 epochs: stop
        break
model.load_state_dict(best_state)
print(f"network: best validation AUC {best_auc:.3f} at epoch {best_epoch}, stopped after epoch {epoch}")

baseline = LogisticRegression(C=0.1, max_iter=2000).fit(X_train.numpy(), y_train.numpy())
print("\ntest AUC (each model scored once):")
print(f"  logistic regression  {roc_auc_score(y_test.numpy(), baseline.predict_proba(X_test.numpy())[:, 1]):.3f}")
print(f"  neural network       {roc_auc_score(y_test.numpy(), churn_prob(model, X_test)):.3f}")

The first line of output shows the best validation AUC and the epoch it came from. It also shows when training stopped. That's 20 epochs after the best one, unless the loop hit its limit of 150 epochs first.

Why score each model only once on the test set? Each time you look at a test score and then change something, the test set quietly turns into another validation set. After enough tries, you'll find a version that does well on those 120 customers by luck. One look keeps the score an honest guess of how the model will do on new customers.

The verdict

The network matches logistic regression but doesn't beat it. It's also harder to explain, slower to train, and has more settings to tune. So the professional call is to ship the simpler model.

Why didn't the network win? There are only 600 customers. And their dozen signals are mostly additive: each one pushes churn risk up or down on its own, without much interplay. Logistic regression is built for that shape. Extra layers have little left to find.

That isn't a failure of deep learning. It's a question of fit, like using a food processor to slice one tomato. Neural networks pull ahead when you have:

• a lot of data, and
• raw, high-dimensional inputs, where the useful features have to be learned. Examples are images, audio, text, and sequences of user events.

"High-dimensional" means each example has a huge number of values. Even a small photo has thousands of pixels. Very large business tables can suit networks too. They often use embeddings for columns with thousands of different values, such as product or store IDs. For ordinary business tables, gradient-boosted trees and regularized linear models from Phase 3 are still hard to beat.

Note: a test that ends with "keep the simple model" is still a useful answer. It saves cost, and the model stays easy to explain to the rest of the team.

What you've built in Phase 4

You started this phase with a single neuron. Since then you've worked through:

• backpropagation by hand, and your own autograd engine;
• PyTorch modules, the training loop, mini-batches and optimizers;
• initialization, normalization, dropout and residual connections;
• convolutions, recurrence and embeddings;
• the habits of sanity-checking and honest comparison.

Phase 5 puts these pieces together into the architecture behind every modern LLM: the transformer.