The same three models in PyTorch
A line, logistic regression and softmax regression fitted both ways: the loss, the gradient and the update opened up, with weight decay and mini-batches.
What it covers6
Why it matters
The models of this phase were fitted with one call, fit. The deep learning phase fits models with a loop you write yourself. The two feel like different worlds, and they are not. A linear model is a set of weights, a forward line, a loss and an update, and fit runs that same loop for you. This lesson fits the same three models both ways, a line, logistic regression and softmax regression, and shows that the numbers come out equal. After it, a PyTorch training loop is the fit you already know, opened up.
What you'll be able to do
- Say what
fitdoes in terms of a loss, a gradient and an update, and write that loop for a linear model in PyTorch - Match each scikit-learn model to its PyTorch layer and loss: a line to MSE, logistic regression to BCEWithLogitsLoss, softmax regression to cross_entropy
- Show that scikit-learn and a PyTorch loop reach the same weights, to three places
- Name the mistake of forgetting
zero_grad(), and say whatweight_decayand aDataLoadercorrespond to in scikit-learn terms
LinearRegression().fit(X, y) and get weights. In PyTorch you write a loop that repeats forward, loss, backward and update. If both fit the same line to the same data, do they reach different weights?Show the answer
No: both minimize the same loss, so they reach the same weights
Least squares has one best line. fit finds it with algebra, and a PyTorch loop finds it by walking down the loss. If the loop runs long enough with a sensible step, both end at the same weights, which is what this lesson shows.
The idea: a model is weights, a forward line, a loss and an update
Purpose. See fit as a loop you can write, so that a PyTorch training loop reads as something you already know.
Every model in this phase has four parts, and fit is the machinery that connects them:
- Weights: the numbers learned from the data.
- A forward line: how the weights and a row of features make a prediction.
- A loss: one number for how wrong the predictions are.
- An update: a step that changes the weights to make the loss smaller.
A line, logistic regression and softmax regression share the forward line, a weighted sum, and differ only in the loss and in what the output is turned into. PyTorch names each part:
| scikit-learn | PyTorch layer | loss | the output is |
|---|---|---|---|
LinearRegression | nn.Linear(d, 1) | F.mse_loss | the prediction itself |
LogisticRegression | nn.Linear(d, 1) | nn.BCEWithLogitsLoss | a score, which the loss turns into a chance |
softmax regression (LogisticRegression with three or more classes) | nn.Linear(d, k) | F.cross_entropy | k scores, one per class |
In plain words: all three are a weighted sum. They differ in how the sum is scored against the truth.
A parameter is one learned number. A layer nn.Linear(d, k) holds d × k weights and k biases.
Show the answer
0.5
The sigmoid of 0 is 1 / (1 + e⁰) = 0.5. A score above 0 means churn is more likely than not. BCEWithLogitsLoss takes the score, the logit, directly and applies the sigmoid inside the loss, which is steadier numerically.
The picture. One training step, with the shape of what flows along each arrow. A batch of n customers with d features goes through the layer and comes out as n predictions. The loss turns them into one number. Backpropagation turns that number into a gradient for every weight, and the update moves the weights.
scikit-learn's fit repeats this until the loss stops falling.
fit connects, and which of them differs between a line and logistic regression?Write a sentence or two, then compare it with a good answer in the lesson.
In code: the same weights, three times
Purpose. Fit a line, logistic regression and softmax regression with scikit-learn and with a PyTorch loop, and compare what they learn.
The browser runs a teaching subset of PyTorch: the same code runs unchanged on a real install, on the CPU, more slowly, and seeded random numbers differ from real PyTorch, so only the structure is compared here, not exact values you would see elsewhere.
First the smallest case, from the lesson on linear regression: three points, (1, 2), (2, 4) and (3, 5). fit finds the line with algebra. A PyTorch loop finds it by gradient descent:
Setup (runs before every cell) · 33 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import numpy as np import pandas as pd import requests import torch import torch.nn as nn import torch.nn.functional as F from sklearn.linear_model import LinearRegression, LogisticRegression, Ridge from sklearn.preprocessing import StandardScaler df = pd.DataFrame(requests.get("https://api.gradient.test/v1/customers").json()["data"]) df["email_open_rate"] = df["email_open_rate"].fillna(df["email_open_rate"].median()) def tensor(a, kind="float"): t = torch.tensor(np.asarray(a)) return t.float() if kind == "float" else t.long() # a line: lifetime value from three columns, everything in z-scores ltv_cols = ["tenure_months", "orders_last_year", "avg_order_value"] X_ltv = StandardScaler().fit_transform(df[ltv_cols]) y_ltv = ((df["lifetime_value"] - df["lifetime_value"].mean()) / df["lifetime_value"].std()).values # logistic regression: churn from four columns churn_cols = ["tenure_months", "orders_last_year", "days_since_last_order", "support_tickets"] X_churn = StandardScaler().fit_transform(df[churn_cols]) y_churn = df["churned"].values # softmax regression: top_category (three classes) from four columns cat_cols = ["orders_last_year", "avg_order_value", "discount_share", "tenure_months"] X_cat = StandardScaler().fit_transform(df[cat_cols]) classes = sorted(df["top_category"].unique()) y_cat = df["top_category"].map({c: i for i, c in enumerate(classes)}).values
x = np.array([[1.0], [2.0], [3.0]]) y = np.array([2.0, 4.0, 5.0]) line = LinearRegression().fit(x, y) print("scikit-learn:", line.coef_.round(3).tolist(), round(float(line.intercept_), 3)) net = nn.Linear(1, 1) opt = torch.optim.SGD(net.parameters(), lr=0.05) xt, yt = tensor(x), tensor(y).reshape(-1, 1) for step in range(3000): opt.zero_grad() loss = F.mse_loss(net(xt), yt) loss.backward() opt.step() print("PyTorch: ", [round(net.weight.item(), 3)], round(net.bias.item(), 3))
Both give a slope of 1.5 and an intercept of 0.667. The four lines inside the loop are the whole of training: clear the old gradients, run the forward line and the loss, run backward to get the gradients, and take a step.
w, grad, lr = 1.0, 4.0, 0.25 w = w - lr * grad print(w)
Show the answer
0.0
The step is 0.25 × 4.0 = 1.0, subtracted from 1.0, leaving 0.0. This is the whole update rule, and optimizer.step() does it for every weight at once.
Worked example: the line on the shop's customers, both ways
Lifetime value from three columns, all in z-scores so the loop converges quickly. Fit LinearRegression, then a PyTorch loop of 300 full-batch steps, and print both weight vectors.
Uses the setup above, 33 lines.
lin = LinearRegression().fit(X_ltv, y_ltv) print("scikit-learn:", lin.coef_.round(3).tolist(), "bias", round(lin.intercept_, 3)) torch.manual_seed(0) net = nn.Linear(3, 1) opt = torch.optim.SGD(net.parameters(), lr=0.1) Xt, yt = tensor(X_ltv), tensor(y_ltv).reshape(-1, 1) for step in range(300): opt.zero_grad() loss = F.mse_loss(net(Xt), yt) loss.backward() opt.step() print("PyTorch: ", net.weight.detach().numpy()[0].astype(float).round(3).tolist(), "bias", round(net.bias.item(), 3))
The two weight vectors agree to three places. The columns are tenure, orders last year and average order value, so each extra standard deviation of orders adds about half a standard deviation to lifetime value. The loop found by walking downhill what scikit-learn found by algebra.
Logistic regression, and a named mistake
The churn model: four columns, one score per customer, and BCEWithLogitsLoss. The scikit-learn side uses C=1e10, a huge C, which makes the default ridge penalty negligible so that both sides minimize the same loss:
Uses the setup above, 33 lines.
logit = LogisticRegression(C=1e10, max_iter=5000).fit(X_churn, y_churn) print("scikit-learn:", logit.coef_.round(3).tolist()[0], logit.intercept_.round(3).tolist()) net = nn.Linear(4, 1) nn.init.zeros_(net.weight) nn.init.zeros_(net.bias) opt = torch.optim.SGD(net.parameters(), lr=1.0) loss_fn = nn.BCEWithLogitsLoss() Xt, yt = tensor(X_churn), tensor(y_churn).reshape(-1, 1) for step in range(2000): opt.zero_grad() loss = loss_fn(net(Xt), yt) loss.backward() opt.step() print("PyTorch: ", net.weight.detach().numpy()[0].astype(float).round(3).tolist(), net.bias.detach().numpy().astype(float).round(3).tolist())
Equal again, to three places. The coefficients are the log-odds of churn for one standard deviation of each column.
The named mistake: forgetting zero_grad(). PyTorch adds each new gradient to the one already stored in the weight, because some algorithms need that. If you never clear it, every step uses the sum of all the gradients so far:
Uses the setup above, 33 lines.
w = torch.tensor([1.0], requires_grad=True) for step in range(3): loss = (w * 2).sum() loss.backward() print("step", step + 1, "gradient stored:", w.grad.tolist())
The true gradient is 2 every time, but the stored value climbs: 2, 4, 6. A loop without zero_grad() takes bigger and bigger, wrong steps.
optimizer.zero_grad(). What does the stored gradient look like after three steps, if every step's own gradient is 5?import torch w = torch.tensor([0.0], requires_grad=True) for step in range(3): (w * 5).sum().backward() print(w.grad.tolist())
Show the answer
[15.0]
Each backward() adds its gradient to w.grad, so after three steps it holds 5 + 5 + 5 = 15. zero_grad() clears it so each step uses only its own gradient.
Fit a line in PyTorch
Write fit_line(X, y, lr, steps). X is an array of shape (n, d) and y has n values. Use nn.Linear(d, 1), F.mse_loss and torch.optim.SGD, and run steps full-batch steps. Return a tuple: the weights as a list of d floats and the bias as a float, each rounded to 3 places.
Uses the setup above, 33 lines.
import numpy as np import torch import torch.nn as nn import torch.nn.functional as F def fit_line(X, y, lr, steps): ...
Find the bug: a loop that never clears
fit_slope(x, y, lr, steps) trains one weight with no bias and returns it. It runs, but the answer is wrong, because the stored gradient grows with every step. Fix it.
Uses the setup above, 33 lines.
import torch def fit_slope(x, y, lr, steps): w = torch.tensor(0.0, requires_grad=True) xt, yt = torch.tensor(x).float(), torch.tensor(y).float() for step in range(steps): loss = ((w * xt - yt) ** 2).mean() loss.backward() with torch.no_grad(): w -= lr * w.grad return round(w.item(), 3)
Show the answer
nn.Linear(d, 3) and F.cross_entropy on the three scores
Three classes need three scores, one per class, and F.cross_entropy applies the softmax and the log loss to them using the integer labels. BCEWithLogitsLoss is for one score and a yes or no label.
Softmax regression for three classes
The same loop with three classes. Each customer's most-bought category is predicted from four columns. nn.Linear(4, 3) gives three scores per customer, and F.cross_entropy takes the scores and the class numbers directly and applies the softmax inside:
Uses the setup above, 33 lines.
soft = LogisticRegression(C=1e10, max_iter=5000).fit(X_cat, y_cat) p_sk = soft.predict_proba(X_cat) net = nn.Linear(4, 3) nn.init.zeros_(net.weight) nn.init.zeros_(net.bias) opt = torch.optim.SGD(net.parameters(), lr=1.0) Xt, yt = tensor(X_cat), tensor(y_cat, "long") for step in range(3000): opt.zero_grad() loss = F.cross_entropy(net(Xt), yt) loss.backward() opt.step() p_torch = torch.softmax(net(Xt), dim=1).detach().numpy() print(classes, "| layer shape", tuple(net.weight.shape)) print("largest gap between the two sets of probabilities:", round(float(abs(p_sk - p_torch).max()), 4)) print("same predicted class for every customer:", bool((p_sk.argmax(axis=1) == p_torch.argmax(axis=1)).all()))
The weights themselves can differ here, because adding the same number to all three classes' scores changes no probability, but the probabilities agree to well under a hundredth, and so does every prediction. The honest result from the multiclass lesson carries over: four columns predict a customer's favorite category only a little better than the most common one.
Write a sentence or two, then compare it with a good answer in the lesson.
In practice: where weight_decay and DataLoader come from
Purpose. Connect the rest of the vocabulary of a PyTorch loop to what you already know, and decide which tool to reach for.
weight_decay is ridge. Ridge regression adds a penalty on the size of the weights. In PyTorch the same pull toward zero is one argument of the optimizer: weight_decay adds a multiple of each weight to its gradient. For a mean squared loss, ridge with penalty alpha equals weight decay wd when alpha = n × wd / 2, where n is the number of rows:
Uses the setup above, 33 lines.
wd = 0.1 n = len(X_ltv) ridge = Ridge(alpha=n * wd / 2).fit(X_ltv, y_ltv) print("Ridge: ", ridge.coef_.round(3).tolist()) net = nn.Linear(3, 1) nn.init.zeros_(net.weight) nn.init.zeros_(net.bias) opt = torch.optim.SGD(net.parameters(), lr=0.1, weight_decay=wd) Xt, yt = tensor(X_ltv), tensor(y_ltv).reshape(-1, 1) for step in range(1000): opt.zero_grad() F.mse_loss(net(Xt), yt).backward() opt.step() print("weight_decay: ", net.weight.detach().numpy()[0].astype(float).round(3).tolist())
The weights are smaller than the unpenalized ones above, and the two methods agree. A small C in LogisticRegression is the same idea, and AdamW applies weight decay in a way that stays separate from the gradient.
DataLoader is the mini-batch loop. The loop above used every row for every step. With more rows, or a model you can't fit in one pass, a DataLoader deals the rows out in shuffled batches, the same mini-batch idea as the lesson on gradient descent:
Uses the setup above, 33 lines.
from torch.utils.data import DataLoader, TensorDataset data = TensorDataset(tensor(X_ltv), tensor(y_ltv).reshape(-1, 1)) loader = DataLoader(data, batch_size=32, shuffle=True) print(len(data), "rows,", len(loader), "batches per pass, batch sizes", sorted({len(xb) for xb, yb in loader}))
One pass through a loader is an epoch. The training loop becomes a loop over epochs with an inner loop over batches, and each batch runs the same four lines.
Show the answer
scikit-learn: a pipeline and a gradient boosted model on the table
A table of 5,000 rows is where scikit-learn is strongest: preprocessing, cross-validation and strong baselines are one line each. Move to PyTorch when the model has layers you want to design, a custom loss, or data that does not fit a table, such as text or images.
Where this turns up in an engineer's week. A scikit-learn model is the baseline, and its loop is hidden. The first time a model needs a layer sklearn does not have, a custom loss, an embedding, or GPU training, the loop is yours to write, and it has the same four lines. Reading a training script is then a matter of finding them: where the gradients are cleared, where the loss is computed, where backward() runs, and where the optimizer steps. Most training bugs live in those lines, the missing zero_grad() most famously.
On the job: before trusting a new training loop, run it on a problem with a known answer. A line through three points should reach 1.5 and 0.667, and if it does not, the loop is wrong.
Goes deeper in
- The training loop: the same four lines on a neural network, with batches and epochs.
- Training with gradient descent: the mini-batch idea in NumPy.
- Regularization: ridge and lasso, and why shrinking weights helps.
Write a sentence or two, then compare it with a good answer in the lesson.
Practice what you just read
- 7 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 8 review cards for spaced repetition
Try a quiz question
Which loss goes with nn.Linear(d, 1) for a yes or no label, taking the raw score rather than a chance?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 3 · about 39 hours of study
Machine learning
Learning from data, and knowing when it worked
50 lessons 1 free
- What machine learning is FreePrograms that learn rules from examples instead of being told them.30 min
- Features and labelsSupervised learning: inputs, targets and predictions.30 min
- Train, validation and test setsHold data back so you can trust your results.38 min
- Preparing dataClean, encode and scale data before a model sees it.40 min
- Linear regressionFit a line, and predict numbers from features.42 min
- Loss functionsOne number that says how wrong the model is.35 min
- Training with gradient descentAdjust weights step by step to reduce the loss.42 min
- Learning rateStep size, convergence and divergence.35 min
- Logistic regressionPredict probabilities for yes/no outcomes.42 min
- Multiclass classificationSoftmax and cross-entropy for more than two classes.42 min
- The same three models in PyTorchA line, logistic regression and softmax regression fitted both ways: the loss, the gradient and the update opened up, with weight decay and mini-batches.35 min
- Precision, recall and F1Accuracy lies; these don't.42 min
- ROC curves and AUCJudge a classifier across every threshold.30 min
- Calibration: can the probability be trusted?Bands of predicted risk against what happened, the Brier score, why naive Bayes is over-confident, and fixing it on rows the model did not fit.30 min
- Overfitting and underfittingMemorizing versus learning.30 min
- RegularizationPenalize complexity so models generalize.40 min
- Bias and varianceThe two ways a model can be wrong.30 min
- Cross-validationReuse limited data for more reliable estimates.42 min
- Imbalanced classes in depthRare classes measured: why rebalancing mostly moves the threshold, correcting inflated probabilities, average precision against AUC, why the count of rare cases matters, and thresholds from costs.45 min
- Naive Bayes: classifying with countsA classifier trained by counting: Bayes' rule over words, the naive assumption, log scores, smoothing for unseen words, why it is overconfident, and when it beats logistic regression.45 min
- Nearest neighborsPredict from the most similar examples.35 min
- Decision treesLearn a flowchart of yes/no questions.42 min
- Random forests and boostingMany weak models voting beat one strong one.42 min
- Gradient boosting in depthBoosting built from scratch: trees fit to the gradient of the loss, depth and interactions, learning rate against number of trees, early stopping, and what XGBoost and LightGBM add.45 min
- Support vector machinesThe widest margin, support vectors, the hinge loss and C, kernels for curved boundaries, and when an SVM fits.40 min
- Recommender systemsShrunk averages, item-to-item collaborative filtering, matrix factorization, and cold start.40 min
- Project: a price modelMultiple regression on log prices, one-hot categories, percentage effects and an extrapolation flag.45 min
- Feature engineeringTurn raw data into signals a model can use.35 min
- The scikit-learn workflowFit, predict and evaluate with the standard ML library.45 min
- Explaining a model: what it uses, and why this predictionPermutation importance on held-back rows, a linear model's contributions for one customer, partial dependence, and what none of them can say.35 min
- Capstone: churn predictionPredict which customers will leave, and measure it properly.45 min
- ClusteringGroup similar items without labels (k-means).35 min
- Clusters of any shape: DBSCAN and hierarchiesWhy k-means needs round clusters, DBSCAN's core points and noise, and hierarchical clustering with its linkages.40 min
- Anomaly detectionMasking, robust scores with the median and MAD, baselines from the past, isolation forests and thresholds from a review budget.40 min
- Dimensionality reductionVisualize and compress high-dimensional data.35 min
- Cleaning messy dataDuplicates, junk rows, text numbers, mixed dates and missing values that mean something.45 min
- Grouping, pivoting and reshapingNamed aggregation, pivot tables, transform, running totals and groups too small to rank.45 min
- Data in pieces: partitions and mergeable summariesSplit, summarize and combine; which statistics merge exactly, merging variances, sketches for the rest, and partitioning well.30 min
- Combining tables with mergeJoin on keys safely, stack tables, and divide by the right denominator.44 min
- Time series with pandasResample, smooth, compare across years, and test whether one series leads another.45 min
- Forecasting: baselines, lag features and backtestsThree baselines, lag features that never see their own future, a walk-forward backtest, and a range around the number.40 min
- Text mining: TF-IDF, search and labelsBag of words and TF-IDF, similarity search and its blind spots, text classifiers and topics, and when LLM labels are worth it.35 min
- Location dataHaversine distances, nearest places, GeoJSON, point in polygon and simple maps.40 min
- Comparing groups: t-testsWelch's t-test, pooled vs average rates, and a control group for before-and-after changes.40 min
- More tests: groups, categories, relationshipsChoose a test, ANOVA by hand, chi-square on counts, Pearson vs Spearman, one-sided tests.40 min
- Project: an exploratory analysis reportFive business questions answered with cleaning, joins, charts and a test, then written up.45 min
- Bandits: explore or exploitLearning from rewards with no labels: a running estimate for each option, why the greedy choice gets stuck, epsilon-greedy and an optimism bonus.35 min
- Decisions in sequence: states, rewards and valueA Markov decision process on a grid world: discounted return, the value of a state, the Bellman equation, and value iteration to the best policy.40 min
- Q-learning: learning values by doingA value for every action, learned from experience with no rules given: the update, a grid world you can watch, and why optimism explores better than chance.40 min
- Policy gradients: learning what to do directlyA policy as probabilities you can nudge: turn up what worked, measure it against a baseline, and see how the same update trains a language model.40 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write