Multiclass classification

Training: the right answer as a list

To train the model, you need past examples where you know the right class. Each right answer is written as a list with a 1 for the true class and 0 everywhere else. A list like that is called a one-hot vector. With the classes in the order apparel, home, accessories:

• a customer who mostly buys apparel is [1, 0, 0];
• one who mostly buys home goods is [0, 1, 0];
• one who mostly buys accessories is [0, 0, 1].

The loss is the cross-entropy from Phase 2. With one right answer, it's $-\log p_{\text{correct}}$: minus the natural log (ln) of the probability the model gave the true class. Say our customer really buys home goods most. The model gave home only 0.245, so the loss is −ln 0.245, about 1.41.

Which way to nudge each weight

Training uses gradient descent, from earlier lessons. It needs a slope for every weight. The slope says how much the loss changes when that weight goes up a little. For softmax with cross-entropy, the slopes have a famously simple form: predicted minus true.

Here it is for our customer, whose true class is home:

A positive gap means "this class got too much probability", so its score gets pushed down. A negative gap means "too little", so its score gets pushed up. Home gets the big push up.

Each weight's slope is its feature value times its class's gap. The customer had 3 orders. So the slope for apparel's orders weight is 3 × 0.665 = 1.995. For home's orders weight it's 3 × −0.755 = −2.265.

With many customers, you average these slopes over all of them. Written as one formula:

There are several symbols here, so take them one at a time:

• $L$ is the loss.
• $\nabla_W L$ is the gradient: a table with one slope for every weight. The triangle, $\nabla$, is said "nabla".
• $n$ is the number of examples.
• $X$ is the table of features, one row per example.
• $X^\top$, said "X transposed", is $X$ flipped so rows become columns.
• $P$ holds the predicted probabilities, one row per example and one column per class.
• $Y$ holds the one-hot truths, in the same shape as $P$.

In plain words: for every weight, multiply each example's feature value by that class's "predicted minus true" gap. Then average over all the examples.

Here's the same calculation in Python, for our one customer:

x = [3, 1]                    # one customer: 3 orders, used a discount
p = [0.665, 0.245, 0.090]     # predicted: apparel, home, accessories
y = [0, 1, 0]                 # one-hot truth: this customer mostly buys home

gap = [pk - yk for pk, yk in zip(p, y)]
print("predicted minus true:", [round(g, 3) for g in gap])
for name, xj in zip(["orders", "discount"], x):
    print(f"slopes for the {name} weights:", [round(xj * g, 3) for g in gap])

The gaps are [0.665, −0.755, 0.09]. The orders weights get slopes of [1.995, −2.265, 0.27]. The discount feature is 1, so its slopes equal the gaps.

A training step moves each weight a little against its slope. With a learning rate of 0.1, apparel's orders weight goes from 0.5 to 0.5 − 0.1 × 1.995 = 0.3005. Home's goes from 0.5 to 0.5 + 0.1 × 2.265 = 0.7265. Next time, this customer's home score will be higher. Repeat that over many steps and many customers, and you have trained the model. The exercise has you build exactly this loop.

Predicting a customer's favorite category

Now try it on bigger data. The practice shop has 600 customers. Each one has a top_category: the kind of product they buy most, apparel, home or accessories. Can four facts about how someone shops predict it? The facts are orders last year, average order value, the share of orders that used a discount, and months as a customer.

The code follows the usual steps from earlier lessons:

• It holds back 30% of the customers as a test set. stratify keeps the mix of categories the same in both parts.
• It puts every feature on the same scale with StandardScaler.
• It fits scikit-learn's LogisticRegression. Give it three classes, and it fits softmax regression for you. (By default it also adds a small penalty that keeps the weights modest. The regularization lesson covers that.)

Then it prints three things. First come the class names. Next is the test accuracy beside the majority baseline. That's the accuracy you'd get by always guessing the most common class. Here that's apparel, a little under half of the customers. Last is a confusion matrix, which the next section explains.

import numpy as np
import pandas as pd
import requests
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

df = pd.DataFrame(requests.get("https://api.gradient.test/v1/customers").json()["data"])
features = ["orders_last_year", "avg_order_value", "discount_share", "tenure_months"]
X_train, X_test, y_train, y_test = train_test_split(df[features], df["top_category"], test_size=0.3, random_state=1, stratify=df["top_category"])

model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)
print("classes:", list(model.classes_))
print("accuracy:", round(model.score(X_test, y_test), 3), " majority baseline:", round(y_test.value_counts(normalize=True).max(), 3))
print(pd.DataFrame(confusion_matrix(y_test, model.predict(X_test), labels=model.classes_),
                   index=[f"true {c}" for c in model.classes_], columns=[f"pred {c}" for c in model.classes_]))

Reading the confusion matrix

A single accuracy number says how often the model is right. It doesn't say what it gets wrong. A confusion matrix does. It's a tally table, like a mail room keeping count of where letters should have gone and where they actually went.

• Each row is a true class.
• Each column is a predicted class.
• Each cell counts the examples with that true class and that prediction.

The diagonal, from top left to bottom right, holds the correct predictions. Every cell off the diagonal is one specific mix-up, such as "really home, but predicted apparel".

classes = ["sales", "support", "billing"]
true = ["billing"] * 5 + ["support"] * 3 + ["sales"] * 2
pred = ["billing"] * 5 + ["billing", "support", "support"] + ["billing", "sales"]

counts = [[0, 0, 0] for _ in classes]       # rows: true class, columns: predicted
for t, p in zip(true, pred):
    counts[classes.index(t)][classes.index(p)] += 1

for c, row in zip(classes, counts):
    print(f"true {c:<8}", row)
correct = sum(t == p for t, p in zip(true, pred))
print("accuracy:", correct / len(true))
print("always 'billing':", true.count("billing") / len(true))

What to look for

• A class that's almost never predicted. Its column is nearly empty. The model has given up on it. Usually that's because the class is rare, or its features overlap with a bigger class.
• Two classes that swap a lot. Their features look alike to the model. Better features, not a bigger model, usually fix that.
• Accuracy next to the baseline. When classes are unbalanced, always guessing the biggest one is already right a large share of the time. An accuracy only means something next to that.

What the customer model shows

Here the model does worse than always guessing apparel. The confusion matrix shows why. It predicts apparel for almost everyone. Order value, order count and discount use barely tell you what people buy.

That's a realistic and useful result. Not every question can be answered from the data you have. The baseline and the confusion matrix tell you so in a minute, before anyone builds a campaign on the predictions. The fix is better features, such as what customers browse or what they bought first. A fancier model won't help.

Other ways to handle many classes

• One-vs-rest: train one yes/no classifier per class, such as "apparel or not?" Then pick the class whose classifier is most confident. This works with any yes/no model.
• Trees and forests handle many classes on their own. They come in the next lessons.
• Language models are softmax classifiers too. Each class is a token in the vocabulary. A huge network computes the scores $\mathbf{z}$, one per token, and softmax turns them into probabilities for the next token.

For marketers: routing inbound messages to sales, support or partnerships is multiclass classification. An LLM with structured output (Phase 1) is often the quickest classifier to build. Either way, this lesson's tools, the confusion matrix and the baseline, are how you judge it.