Training with gradient descent

import numpy as np

x = [1, 2]          # the feature, one value per row in the batch
y = [3, 5]          # the true values
w, b = 0.0, 0.0     # the weight and the intercept, starting at zero
lr = 0.05           # the learning rate
n = len(x)          # rows in this batch

errors = [(w * xi + b) - yi for xi, yi in zip(x, y)]
grad_w = 2 / n * sum(e * xi for e, xi in zip(errors, x))
grad_b = 2 / n * sum(e * 1 for e in errors)
print("errors:", errors)
print("gradient:", grad_w, grad_b)

w = w - lr * grad_w
b = b - lr * grad_b
print("new weight and intercept:", w, b)

X = np.array([[1.0, 1.0], [2.0, 1.0]])   # the feature column, then the column of ones
print("same gradient in NumPy:", 2 / n * X.T @ (X @ np.zeros(2) - np.array(y)))

w = 3.0
grad = -4.0
lr = 0.25
w -= lr * grad
print(w)

From scratch on the customer data

Now the full loop on a real task. The shop has 600 customers. The goal is to predict each one's lifetime value (their total spend so far) from four features. Those are orders last year, average order value, months as a customer, and loyalty-club membership.

Before the loop, the code does two set-up steps:

• It standardizes each feature, as in the data preparation lesson. Each one is rescaled so its average is 0 and its spread is 1.
• It adds the column of 1s for the intercept.

Then it trains for 30 epochs with batches of 32 and a learning rate of 0.05. The 600 rows make 18 full batches plus one of 24, so 19 updates per epoch and 570 in all. A fixed seed makes the shuffling the same on every run. At the end, it prints the weights next to the exact answer from np.linalg.lstsq ("least squares").

import numpy as np
import pandas as pd
import requests
import matplotlib.pyplot as plt

df = pd.DataFrame(requests.get("https://api.gradient.test/v1/customers").json()["data"])
features = ["orders_last_year", "avg_order_value", "tenure_months", "trail_club"]
X_raw = df[features].to_numpy(dtype=float)
mean, std = X_raw.mean(axis=0), X_raw.std(axis=0)
X = np.column_stack([(X_raw - mean) / std, np.ones(len(df))])    # standardized features + bias column
y = df["lifetime_value"].to_numpy()

rng = np.random.default_rng(0)
w = np.zeros(X.shape[1])
lr, batch, losses = 0.05, 32, []
for epoch in range(30):
    order = rng.permutation(len(y))
    for start in range(0, len(y), batch):
        idx = order[start:start + batch]
        grad = 2 / len(idx) * X[idx].T @ (X[idx] @ w - y[idx])
        w -= lr * grad
    losses.append(np.mean((X @ w - y) ** 2))

exact, *_ = np.linalg.lstsq(X, y, rcond=None)
print("gradient descent:", np.round(w, 1).tolist())
print("exact solution:  ", np.round(exact, 1).tolist())
plt.plot(losses)
plt.yscale("log")
plt.xlabel("epoch")
plt.ylabel("mean squared error")

After 30 epochs, every weight is within about $10 of the exact answer. It doesn't land exactly on it. Each batch gives a slightly different gradient, so the weights keep jiggling around the best spot. A learning rate that shrinks late in training closes that gap. That's the next lesson.

The chart is the loss curve, one point per epoch. The loss drops in the first epoch or two, then flattens into a slightly jagged line. That's the classic shape.

Reading loss curves

The loss curve is the first thing to check after any training run. Here's what its common shapes usually mean:

"Converged" means the weights have settled near the best spot. "Validation" data is data held back from training, as in the lesson on train, validation and test sets.

Always plot the loss on held-out data too. A model is only getting better if that curve falls. A falling training loss can also mean the model is memorizing its training rows.

When do you need gradient descent?

For linear regression on 600 customers, you'd use the exact solution in practice. What you keep from this lesson is the loop itself. A PyTorch training script, which you'll write in Phase 4, has the same shape. PyTorch is a popular deep-learning library. Look for the familiar parts: epochs, shuffled batches, a loss, the gradient and the update.

for epoch in range(epochs):
    for xb, yb in loader:                 # shuffled mini-batches
        loss = ((model(xb) - yb) ** 2).mean()
        optimizer.zero_grad()
        loss.backward()                   # the gradient, by backpropagation
        optimizer.step()                  # w ← w − lr × gradient (or Adam's version)

The loader hands out shuffled mini-batches. The line loss.backward() works out the gradient for you, using a method called backpropagation. The line optimizer.step() applies the update. Adam, mentioned in the table, is a smarter version of that update and the usual choice for neural networks.