Index Ai Academy
Phase 3 · Models

The same three models in PyTorch

A line, logistic regression and softmax regression fitted both ways: the loss, the gradient and the update opened up, with weight decay and mini-batches.

About 35 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

The models of this phase were fitted with one call, fit. The deep learning phase fits models with a loop you write yourself. The two feel like different worlds, and they are not. A linear model is a set of weights, a forward line, a loss and an update, and fit runs that same loop for you. This lesson fits the same three models both ways, a line, logistic regression and softmax regression, and shows that the numbers come out equal. After it, a PyTorch training loop is the fit you already know, opened up.

What you'll be able to do

  • Say what fit does in terms of a loss, a gradient and an update, and write that loop for a linear model in PyTorch
  • Match each scikit-learn model to its PyTorch layer and loss: a line to MSE, logistic regression to BCEWithLogitsLoss, softmax regression to cross_entropy
  • Show that scikit-learn and a PyTorch loop reach the same weights, to three places
  • Name the mistake of forgetting zero_grad(), and say what weight_decay and a DataLoader correspond to in scikit-learn terms
Quick check
A guess first. In scikit-learn you call LinearRegression().fit(X, y) and get weights. In PyTorch you write a loop that repeats forward, loss, backward and update. If both fit the same line to the same data, do they reach different weights?
  1. Yes: PyTorch loops always end at somewhat worse weights
  2. It depends on the random seed the loop starts from
  3. Yes: scikit-learn minimizes a different loss from the loop
  4. No: both minimize the same loss, so they reach the same weights
Show the answer

No: both minimize the same loss, so they reach the same weights

Least squares has one best line. fit finds it with algebra, and a PyTorch loop finds it by walking down the loss. If the loop runs long enough with a sensible step, both end at the same weights, which is what this lesson shows.

The idea: a model is weights, a forward line, a loss and an update

Purpose. See fit as a loop you can write, so that a PyTorch training loop reads as something you already know.

Every model in this phase has four parts, and fit is the machinery that connects them:

  • Weights: the numbers learned from the data.
  • A forward line: how the weights and a row of features make a prediction.
  • A loss: one number for how wrong the predictions are.
  • An update: a step that changes the weights to make the loss smaller.

A line, logistic regression and softmax regression share the forward line, a weighted sum, and differ only in the loss and in what the output is turned into. PyTorch names each part:

scikit-learnPyTorch layerlossthe output is
LinearRegressionnn.Linear(d, 1)F.mse_lossthe prediction itself
LogisticRegressionnn.Linear(d, 1)nn.BCEWithLogitsLossa score, which the loss turns into a chance
softmax regression (LogisticRegression with three or more classes)nn.Linear(d, k)F.cross_entropyk scores, one per class

In plain words: all three are a weighted sum. They differ in how the sum is scored against the truth.

A parameter is one learned number. A layer nn.Linear(d, k) holds d × k weights and k biases.

Quick check
From the lesson on logistic regression: the model's score for a customer is a weighted sum, and the chance of churning comes from the sigmoid of that score. A score of 0 gives what chance?
  1. 0
  2. 1
  3. 0.25
  4. 0.5
Show the answer

0.5

The sigmoid of 0 is 1 / (1 + e⁰) = 0.5. A score above 0 means churn is more likely than not. BCEWithLogitsLoss takes the score, the logit, directly and applies the sigmoid inside the loss, which is steadier numerically.

The picture. One training step, with the shape of what flows along each arrow. A batch of n customers with d features goes through the layer and comes out as n predictions. The loss turns them into one number. Backpropagation turns that number into a gradient for every weight, and the update moves the weights.

Try it now

scikit-learn's fit repeats this until the loss stops falling.

In your own words
In your own words: what are the four parts of a model that fit connects, and which of them differs between a line and logistic regression?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: the same weights, three times

Purpose. Fit a line, logistic regression and softmax regression with scikit-learn and with a PyTorch loop, and compare what they learn.

The browser runs a teaching subset of PyTorch: the same code runs unchanged on a real install, on the CPU, more slowly, and seeded random numbers differ from real PyTorch, so only the structure is compared here, not exact values you would see elsewhere.

First the smallest case, from the lesson on linear regression: three points, (1, 2), (2, 4) and (3, 5). fit finds the line with algebra. A PyTorch loop finds it by gradient descent:

Setup (runs before every cell) · 33 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import numpy as np
import pandas as pd
import requests
import torch
import torch.nn as nn
import torch.nn.functional as F
from sklearn.linear_model import LinearRegression, LogisticRegression, Ridge
from sklearn.preprocessing import StandardScaler

df = pd.DataFrame(requests.get("https://api.gradient.test/v1/customers").json()["data"])
df["email_open_rate"] = df["email_open_rate"].fillna(df["email_open_rate"].median())


def tensor(a, kind="float"):
    t = torch.tensor(np.asarray(a))
    return t.float() if kind == "float" else t.long()


# a line: lifetime value from three columns, everything in z-scores
ltv_cols = ["tenure_months", "orders_last_year", "avg_order_value"]
X_ltv = StandardScaler().fit_transform(df[ltv_cols])
y_ltv = ((df["lifetime_value"] - df["lifetime_value"].mean())
         / df["lifetime_value"].std()).values
# logistic regression: churn from four columns
churn_cols = ["tenure_months", "orders_last_year", "days_since_last_order",
              "support_tickets"]
X_churn = StandardScaler().fit_transform(df[churn_cols])
y_churn = df["churned"].values
# softmax regression: top_category (three classes) from four columns
cat_cols = ["orders_last_year", "avg_order_value", "discount_share", "tenure_months"]
X_cat = StandardScaler().fit_transform(df[cat_cols])
classes = sorted(df["top_category"].unique())
y_cat = df["top_category"].map({c: i for i, c in enumerate(classes)}).values
x = np.array([[1.0], [2.0], [3.0]])
y = np.array([2.0, 4.0, 5.0])
line = LinearRegression().fit(x, y)
print("scikit-learn:", line.coef_.round(3).tolist(), round(float(line.intercept_), 3))

net = nn.Linear(1, 1)
opt = torch.optim.SGD(net.parameters(), lr=0.05)
xt, yt = tensor(x), tensor(y).reshape(-1, 1)
for step in range(3000):
    opt.zero_grad()
    loss = F.mse_loss(net(xt), yt)
    loss.backward()
    opt.step()
print("PyTorch:     ", [round(net.weight.item(), 3)], round(net.bias.item(), 3))

Both give a slope of 1.5 and an intercept of 0.667. The four lines inside the loop are the whole of training: clear the old gradients, run the forward line and the loss, run backward to get the gradients, and take a step.

Quick check
A weight is 1.0 with a gradient of 4.0 and a learning rate of 0.25. What does this print?
w, grad, lr = 1.0, 4.0, 0.25
w = w - lr * grad
print(w)
  1. 0.0
  2. -3.0
  3. 1.0
  4. 2.0
Show the answer

0.0

The step is 0.25 × 4.0 = 1.0, subtracted from 1.0, leaving 0.0. This is the whole update rule, and optimizer.step() does it for every weight at once.

Worked example

Worked example: the line on the shop's customers, both ways

Lifetime value from three columns, all in z-scores so the loop converges quickly. Fit LinearRegression, then a PyTorch loop of 300 full-batch steps, and print both weight vectors.

Uses the setup above, 33 lines.

lin = LinearRegression().fit(X_ltv, y_ltv)
print("scikit-learn:", lin.coef_.round(3).tolist(), "bias", round(lin.intercept_, 3))

torch.manual_seed(0)
net = nn.Linear(3, 1)
opt = torch.optim.SGD(net.parameters(), lr=0.1)
Xt, yt = tensor(X_ltv), tensor(y_ltv).reshape(-1, 1)
for step in range(300):
    opt.zero_grad()
    loss = F.mse_loss(net(Xt), yt)
    loss.backward()
    opt.step()
print("PyTorch:     ", net.weight.detach().numpy()[0].astype(float).round(3).tolist(),
      "bias", round(net.bias.item(), 3))
In the lesson you can change this and run it.

The two weight vectors agree to three places. The columns are tenure, orders last year and average order value, so each extra standard deviation of orders adds about half a standard deviation to lifetime value. The loop found by walking downhill what scikit-learn found by algebra.

Logistic regression, and a named mistake

The churn model: four columns, one score per customer, and BCEWithLogitsLoss. The scikit-learn side uses C=1e10, a huge C, which makes the default ridge penalty negligible so that both sides minimize the same loss:

Uses the setup above, 33 lines.

logit = LogisticRegression(C=1e10, max_iter=5000).fit(X_churn, y_churn)
print("scikit-learn:", logit.coef_.round(3).tolist()[0],
      logit.intercept_.round(3).tolist())

net = nn.Linear(4, 1)
nn.init.zeros_(net.weight)
nn.init.zeros_(net.bias)
opt = torch.optim.SGD(net.parameters(), lr=1.0)
loss_fn = nn.BCEWithLogitsLoss()
Xt, yt = tensor(X_churn), tensor(y_churn).reshape(-1, 1)
for step in range(2000):
    opt.zero_grad()
    loss = loss_fn(net(Xt), yt)
    loss.backward()
    opt.step()
print("PyTorch:     ", net.weight.detach().numpy()[0].astype(float).round(3).tolist(),
      net.bias.detach().numpy().astype(float).round(3).tolist())

Equal again, to three places. The coefficients are the log-odds of churn for one standard deviation of each column.

The named mistake: forgetting zero_grad(). PyTorch adds each new gradient to the one already stored in the weight, because some algorithms need that. If you never clear it, every step uses the sum of all the gradients so far:

Uses the setup above, 33 lines.

w = torch.tensor([1.0], requires_grad=True)
for step in range(3):
    loss = (w * 2).sum()
    loss.backward()
    print("step", step + 1, "gradient stored:", w.grad.tolist())

The true gradient is 2 every time, but the stored value climbs: 2, 4, 6. A loop without zero_grad() takes bigger and bigger, wrong steps.

Quick check
A training loop forgets optimizer.zero_grad(). What does the stored gradient look like after three steps, if every step's own gradient is 5?
import torch

w = torch.tensor([0.0], requires_grad=True)
for step in range(3):
    (w * 5).sum().backward()
print(w.grad.tolist())
  1. [15.0]
  2. [0.0]
  3. [5.0]
  4. [3.0]
Show the answer

[15.0]

Each backward() adds its gradient to w.grad, so after three steps it holds 5 + 5 + 5 = 15. zero_grad() clears it so each step uses only its own gradient.

Drill

Fit a line in PyTorch

Write fit_line(X, y, lr, steps). X is an array of shape (n, d) and y has n values. Use nn.Linear(d, 1), F.mse_loss and torch.optim.SGD, and run steps full-batch steps. Return a tuple: the weights as a list of d floats and the bias as a float, each rounded to 3 places.

Uses the setup above, 33 lines.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F


def fit_line(X, y, lr, steps):
    ...
In the lesson, your code is checked as you write it.
Drill

Find the bug: a loop that never clears

fit_slope(x, y, lr, steps) trains one weight with no bias and returns it. It runs, but the answer is wrong, because the stored gradient grows with every step. Fix it.

Uses the setup above, 33 lines.

import torch


def fit_slope(x, y, lr, steps):
    w = torch.tensor(0.0, requires_grad=True)
    xt, yt = torch.tensor(x).float(), torch.tensor(y).float()
    for step in range(steps):
        loss = ((w * xt - yt) ** 2).mean()
        loss.backward()
        with torch.no_grad():
            w -= lr * w.grad
    return round(w.item(), 3)
In the lesson, your code is checked as you write it.
Quick check
A model predicts one of three classes, and the labels are the integers 0, 1 and 2. Which pair is right?
  1. nn.Linear(d, 1) and nn.BCEWithLogitsLoss
  2. nn.Linear(d, 3) and F.mse_loss on the three scores
  3. nn.Linear(d, 3) and F.cross_entropy on the three scores
  4. nn.Linear(d, 1) and F.cross_entropy
Show the answer

nn.Linear(d, 3) and F.cross_entropy on the three scores

Three classes need three scores, one per class, and F.cross_entropy applies the softmax and the log loss to them using the integer labels. BCEWithLogitsLoss is for one score and a yes or no label.

Softmax regression for three classes

The same loop with three classes. Each customer's most-bought category is predicted from four columns. nn.Linear(4, 3) gives three scores per customer, and F.cross_entropy takes the scores and the class numbers directly and applies the softmax inside:

Uses the setup above, 33 lines.

soft = LogisticRegression(C=1e10, max_iter=5000).fit(X_cat, y_cat)
p_sk = soft.predict_proba(X_cat)

net = nn.Linear(4, 3)
nn.init.zeros_(net.weight)
nn.init.zeros_(net.bias)
opt = torch.optim.SGD(net.parameters(), lr=1.0)
Xt, yt = tensor(X_cat), tensor(y_cat, "long")
for step in range(3000):
    opt.zero_grad()
    loss = F.cross_entropy(net(Xt), yt)
    loss.backward()
    opt.step()
p_torch = torch.softmax(net(Xt), dim=1).detach().numpy()

print(classes, "| layer shape", tuple(net.weight.shape))
print("largest gap between the two sets of probabilities:",
      round(float(abs(p_sk - p_torch).max()), 4))
print("same predicted class for every customer:",
      bool((p_sk.argmax(axis=1) == p_torch.argmax(axis=1)).all()))

The weights themselves can differ here, because adding the same number to all three classes' scores changes no probability, but the probabilities agree to well under a hundredth, and so does every prediction. The honest result from the multiclass lesson carries over: four columns predict a customer's favorite category only a little better than the most common one.

In your own words
What are the four lines inside every PyTorch training step, and what does each one do?

Write a sentence or two, then compare it with a good answer in the lesson.

In practice: where weight_decay and DataLoader come from

Purpose. Connect the rest of the vocabulary of a PyTorch loop to what you already know, and decide which tool to reach for.

weight_decay is ridge. Ridge regression adds a penalty on the size of the weights. In PyTorch the same pull toward zero is one argument of the optimizer: weight_decay adds a multiple of each weight to its gradient. For a mean squared loss, ridge with penalty alpha equals weight decay wd when alpha = n × wd / 2, where n is the number of rows:

Uses the setup above, 33 lines.

wd = 0.1
n = len(X_ltv)
ridge = Ridge(alpha=n * wd / 2).fit(X_ltv, y_ltv)
print("Ridge:        ", ridge.coef_.round(3).tolist())

net = nn.Linear(3, 1)
nn.init.zeros_(net.weight)
nn.init.zeros_(net.bias)
opt = torch.optim.SGD(net.parameters(), lr=0.1, weight_decay=wd)
Xt, yt = tensor(X_ltv), tensor(y_ltv).reshape(-1, 1)
for step in range(1000):
    opt.zero_grad()
    F.mse_loss(net(Xt), yt).backward()
    opt.step()
print("weight_decay: ", net.weight.detach().numpy()[0].astype(float).round(3).tolist())

The weights are smaller than the unpenalized ones above, and the two methods agree. A small C in LogisticRegression is the same idea, and AdamW applies weight decay in a way that stays separate from the gradient.

DataLoader is the mini-batch loop. The loop above used every row for every step. With more rows, or a model you can't fit in one pass, a DataLoader deals the rows out in shuffled batches, the same mini-batch idea as the lesson on gradient descent:

Uses the setup above, 33 lines.

from torch.utils.data import DataLoader, TensorDataset

data = TensorDataset(tensor(X_ltv), tensor(y_ltv).reshape(-1, 1))
loader = DataLoader(data, batch_size=32, shuffle=True)
print(len(data), "rows,", len(loader), "batches per pass, batch sizes",
      sorted({len(xb) for xb, yb in loader}))

One pass through a loader is an epoch. The training loop becomes a loop over epochs with an inner loop over batches, and each batch runs the same four lines.

Quick check
A team has 5,000 rows and 12 columns of customer data, and wants to predict churn. Which tool should they reach for first?
  1. A PyTorch loop with a custom loss function written by hand
  2. None: that is too little data to build any useful model
  3. A PyTorch loop with a neural network designed for the table
  4. scikit-learn: a pipeline and a gradient boosted model on the table
Show the answer

scikit-learn: a pipeline and a gradient boosted model on the table

A table of 5,000 rows is where scikit-learn is strongest: preprocessing, cross-validation and strong baselines are one line each. Move to PyTorch when the model has layers you want to design, a custom loss, or data that does not fit a table, such as text or images.

Where this turns up in an engineer's week. A scikit-learn model is the baseline, and its loop is hidden. The first time a model needs a layer sklearn does not have, a custom loss, an embedding, or GPU training, the loop is yours to write, and it has the same four lines. Reading a training script is then a matter of finding them: where the gradients are cleared, where the loss is computed, where backward() runs, and where the optimizer steps. Most training bugs live in those lines, the missing zero_grad() most famously.

On the job: before trusting a new training loop, run it on a problem with a known answer. A line through three points should reach 1.5 and 0.667, and if it does not, the loop is wrong.

Goes deeper in

In your own words
A colleague's PyTorch loop gives a different line from scikit-learn's on the same data. What would you check?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 7 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 8 review cards for spaced repetition

Try a quiz question

Which loss goes with nn.Linear(d, 1) for a yes or no label, taking the raw score rather than a chance?

  1. nn.BCEWithLogitsLoss
  2. F.mse_loss
  3. F.cross_entropy
  4. nn.L1Loss

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 3 · about 39 hours of study

Machine learning

Learning from data, and knowing when it worked 50 lessons 1 free
  1. What machine learning is FreePrograms that learn rules from examples instead of being told them.30 min
  2. Features and labelsSupervised learning: inputs, targets and predictions.30 min
  3. Train, validation and test setsHold data back so you can trust your results.38 min
  4. Preparing dataClean, encode and scale data before a model sees it.40 min
  5. Linear regressionFit a line, and predict numbers from features.42 min
  6. Loss functionsOne number that says how wrong the model is.35 min
  7. Training with gradient descentAdjust weights step by step to reduce the loss.42 min
  8. Learning rateStep size, convergence and divergence.35 min
  9. Logistic regressionPredict probabilities for yes/no outcomes.42 min
  10. Multiclass classificationSoftmax and cross-entropy for more than two classes.42 min
  11. The same three models in PyTorchA line, logistic regression and softmax regression fitted both ways: the loss, the gradient and the update opened up, with weight decay and mini-batches.35 min
  12. Precision, recall and F1Accuracy lies; these don't.42 min
  13. ROC curves and AUCJudge a classifier across every threshold.30 min
  14. Calibration: can the probability be trusted?Bands of predicted risk against what happened, the Brier score, why naive Bayes is over-confident, and fixing it on rows the model did not fit.30 min
  15. Overfitting and underfittingMemorizing versus learning.30 min
  16. RegularizationPenalize complexity so models generalize.40 min
  17. Bias and varianceThe two ways a model can be wrong.30 min
  18. Cross-validationReuse limited data for more reliable estimates.42 min
  19. Imbalanced classes in depthRare classes measured: why rebalancing mostly moves the threshold, correcting inflated probabilities, average precision against AUC, why the count of rare cases matters, and thresholds from costs.45 min
  20. Naive Bayes: classifying with countsA classifier trained by counting: Bayes' rule over words, the naive assumption, log scores, smoothing for unseen words, why it is overconfident, and when it beats logistic regression.45 min
  21. Nearest neighborsPredict from the most similar examples.35 min
  22. Decision treesLearn a flowchart of yes/no questions.42 min
  23. Random forests and boostingMany weak models voting beat one strong one.42 min
  24. Gradient boosting in depthBoosting built from scratch: trees fit to the gradient of the loss, depth and interactions, learning rate against number of trees, early stopping, and what XGBoost and LightGBM add.45 min
  25. Support vector machinesThe widest margin, support vectors, the hinge loss and C, kernels for curved boundaries, and when an SVM fits.40 min
  26. Recommender systemsShrunk averages, item-to-item collaborative filtering, matrix factorization, and cold start.40 min
  27. Project: a price modelMultiple regression on log prices, one-hot categories, percentage effects and an extrapolation flag.45 min
  28. Feature engineeringTurn raw data into signals a model can use.35 min
  29. The scikit-learn workflowFit, predict and evaluate with the standard ML library.45 min
  30. Explaining a model: what it uses, and why this predictionPermutation importance on held-back rows, a linear model's contributions for one customer, partial dependence, and what none of them can say.35 min
  31. Capstone: churn predictionPredict which customers will leave, and measure it properly.45 min
  32. ClusteringGroup similar items without labels (k-means).35 min
  33. Clusters of any shape: DBSCAN and hierarchiesWhy k-means needs round clusters, DBSCAN's core points and noise, and hierarchical clustering with its linkages.40 min
  34. Anomaly detectionMasking, robust scores with the median and MAD, baselines from the past, isolation forests and thresholds from a review budget.40 min
  35. Dimensionality reductionVisualize and compress high-dimensional data.35 min
  36. Cleaning messy dataDuplicates, junk rows, text numbers, mixed dates and missing values that mean something.45 min
  37. Grouping, pivoting and reshapingNamed aggregation, pivot tables, transform, running totals and groups too small to rank.45 min
  38. Data in pieces: partitions and mergeable summariesSplit, summarize and combine; which statistics merge exactly, merging variances, sketches for the rest, and partitioning well.30 min
  39. Combining tables with mergeJoin on keys safely, stack tables, and divide by the right denominator.44 min
  40. Time series with pandasResample, smooth, compare across years, and test whether one series leads another.45 min
  41. Forecasting: baselines, lag features and backtestsThree baselines, lag features that never see their own future, a walk-forward backtest, and a range around the number.40 min
  42. Text mining: TF-IDF, search and labelsBag of words and TF-IDF, similarity search and its blind spots, text classifiers and topics, and when LLM labels are worth it.35 min
  43. Location dataHaversine distances, nearest places, GeoJSON, point in polygon and simple maps.40 min
  44. Comparing groups: t-testsWelch's t-test, pooled vs average rates, and a control group for before-and-after changes.40 min
  45. More tests: groups, categories, relationshipsChoose a test, ANOVA by hand, chi-square on counts, Pearson vs Spearman, one-sided tests.40 min
  46. Project: an exploratory analysis reportFive business questions answered with cleaning, joins, charts and a test, then written up.45 min
  47. Bandits: explore or exploitLearning from rewards with no labels: a running estimate for each option, why the greedy choice gets stuck, epsilon-greedy and an optimism bonus.35 min
  48. Decisions in sequence: states, rewards and valueA Markov decision process on a grid world: discounted return, the value of a state, the Bellman equation, and value iteration to the best policy.40 min
  49. Q-learning: learning values by doingA value for every action, learned from experience with no rules given: the update, a grid world you can watch, and why optimism explores better than chance.40 min
  50. Policy gradients: learning what to do directlyA policy as probabilities you can nudge: turn up what worked, measure it against a baseline, and see how the same update trains a language model.40 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write