Linear regression is the simplest model that is trained, tested and used exactly the way large models are. It predicts a number from a number, tents from temperature, with a straight line, and "training" means choosing the line's slope and intercept from data. Nearly every idea of the last four modules meets here: the line from lesson 21, covariance and variance from lessons 94 and 97, gradient descent from lesson 79, solving equations from lesson 60, and the matrix product from lesson 56. They all arrive at the same line, which makes this lesson a good place to see that they're one subject. And the most important habit in machine learning starts here too: judging a model on data it has never seen, not on the data it was trained on, because the error on training data is almost always too flattering.
What you'll be able to do
Measure how well a line fits with residuals and the sum of their squares
Find the least-squares line by formula: w = cov(x, y)/var(x) and b = ȳ − w·x̄
Get the same line three ways in Python: the formula, gradient descent, and solving a small system
Fit on some days and judge on days held back, in the matrix form ŷ = Xw, and train the same line as a PyTorch layer with nn.Linear and optim.SGD
Quick check
Have a guess before you start. Five points don't lie on one line. What would "the best line" through them make as small as possible?
The distance between the first and last point
The total of the squared vertical misses
The number of points it touches
Its slope
Show the answer
The total of the squared vertical misses
Least squares measures each point's vertical miss, squares it, and adds them up; the best line has the smallest total.
The math: the line that misses the least
Purpose. Choose the slope and intercept that fit the points best.
Residuals. A line ŷ = wx + b predicts ŷ ("y hat") for each x. A point's residual is how far the line misses it: y − ŷ. Five days, with x the day and y the rentals: (1, 3), (2, 5), (3, 4), (4, 7), (5, 8). The line ŷ = 1.2x + 1.8 predicts 3, 4.2, 5.4, 6.6 and 7.8, so the residuals are 0, 0.8, −1.4, 0.4 and 0.2.
Squared residuals. Some residuals are positive and some negative, so, as with variance in lesson 97, they're squared before adding: 0 + 0.64 + 1.96 + 0.16 + 0.04 = 2.8. That total is the sum of squared residuals, and divided by the number of points it's the mean squared error, 0.56, lesson 79's loss. The least-squares line is the one with the smallest total, and 1.2x + 1.8 is it: no other line through these points gets below 2.8.
Quick check
From lesson 94: for x = (1, 2, 3, 4, 5) and y = (3, 5, 4, 7, 8), the covariance is 2.4 and the variance of x is 2. What sign does that say the best line's slope has?
Zero
Negative
Positive: y tends to rise with x
It can't be told without fitting
Show the answer
Positive: y tends to rise with x
The slope is cov(x, y) ÷ var(x), and a variance is never negative, so the slope takes the covariance's sign.
The formula. Setting the slopes of the squared-error loss to zero (lesson 77's partial derivatives, one for w and one for b) gives two short rules. Step through them for the five days.
The picture. The drawing builds the line the way the rules do. First the middle point, (x̄, ȳ), where dashed lines at the two means cross. Then a rectangle from the middle point out to each point: its area is that point's (x − x̄)(y − ȳ), the product the slope rule adds up. Then the line through the middle point, then where it crosses x = 0. Last, each miss grows into a square, and the squares' areas are what least squares makes as small as it can.
Try it now
Then set the y values to 3, 5, 4, 7, 12 and press Use these numbers: that one far point tilts the whole line toward it, because a miss counts by its square.
Symbols and formula. The same rules in lesson 94's words:
In plain words: the slope is how much x and y move together, per unit of x's own spread; the line passes through the middle point; and R² is the share of y's spread the line explains.
For the five days, cov = 2.4 and var = 2, so w = 1.2, and b = 5.4 − 1.2 × 3 = 1.8. How good is the fit? The spread of y around its own mean adds to 17.2; the line leaves 2.8 of it, so R² = 1 − 2.8/17.2 = 0.84: the line explains 84% of the ups and downs. R² = 1 is a perfect fit; R² = 0 is no better than always guessing the mean.
Try to beat it. The chart below has points of its own, with each miss drawn from the point to the line. Move the slope and the intercept to make the mean squared error as small as you can, then press Best fit and compare.
Quick check
A line's predictions for three points miss by 1, −2 and 1. What is the sum of squared residuals?
2
6
4
0
Show the answer
6
1 + 4 + 1 = 6. The plain residuals add to 0, which is exactly why they're squared first.
In your own words
In your own words: why are residuals squared before adding, rather than just added?
Write a sentence or two, then compare it with a good answer in the lesson.
In Python: three roads to one line
Purpose. Get the same line by formula, by gradient descent and by solving a system, so each checks the others.
import numpy as np
x = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
y = np.array([3.0, 5.0, 4.0, 7.0, 8.0])
dx = x - x.mean()
w = np.sum(dx * (y - y.mean())) / np.sum(dx ** 2)
b = y.mean() - w * x.mean()
print("formula:", round(w, 4), round(b, 4))
wg, bg = 0.0, 0.0for step inrange(2000):
e = wg * x + bg - y
wg = wg - 0.05 * 2 * np.mean(e * x)
bg = bg - 0.05 * 2 * np.mean(e)
print("descent:", round(wg, 4), round(bg, 4))
A = np.array([[np.sum(x ** 2), np.sum(x)], [np.sum(x), 5.0]])
r = np.array([np.sum(x * y), np.sum(y)])
print("solve:", np.linalg.solve(A, r).round(4).tolist())
It runs right here in your browser, with nothing to install.
All three print 1.2 and 1.8. The formula is quickest; descent is lesson 79's loop and works for any loss; the system is the two equations "both slopes are 0" written out, which lesson 60 solved by elimination.
Code flow. Step through two of the roads one line at a time: the formula, then the system. dx and dy are the math part's centered columns, the same rows of boxes as its flow. A and r hold the system's two equations as a grid and a column, and np.linalg.solve gives back w and b in one call. The third road, descent, is the loop in the cell above: 2,000 passes, too many to step through.
The line in a plot. Draw the five days and the line the roads found. plt.plot(x, y, "o") draws each point as a dot without joining them, and the second plt.plot draws the line through the five predictions.
import numpy as np
import matplotlib.pyplot as plt
x = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
y = np.array([3.0, 5.0, 4.0, 7.0, 8.0])
plt.plot(x, y, "o")
plt.plot(x, 1.2 * x + 1.8)
plt.xlabel("day")
plt.ylabel("rentals")
plt.show()
Quick check
Read the plot. Which day's dot sits furthest from the line?
Day 1: it's the lowest dot
Day 5: it's the highest dot
Day 3: 4 rented, and the line says 5.4
Day 2: 5 rented, and the line says 4.2
Show the answer
Day 3: 4 rented, and the line says 5.4
Day 3's dot is 1.4 below the line, the biggest miss of the five. Squared, that's 1.96 of the 2.8 total: more than the other four days together.
A named mistake: judging a model on its own training data. A line fitted to some points always looks good on those points: it was chosen to. With enough freedom, say a wiggly curve with as many settings as points, the training error can reach 0 while predictions for new x's are wild. The honest test is on points the fit never saw. Keep some data back before fitting, fit on the rest, and report the error on the held-back part.
Worked example
Worked example: the matrix form ŷ = Xw
Put a column of ones next to x, so one matrix product makes every prediction, and let NumPy find the best w and b.
import numpy as np
x = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
y = np.array([3.0, 5.0, 4.0, 7.0, 8.0])
X = np.column_stack([x, np.ones(5)])
wb = np.linalg.lstsq(X, y, rcond=None)[0]
pred = X @ wb
print(wb.round(4).tolist())
print(pred.round(2).tolist(), round(float(np.sum((y - pred) ** 2)), 2))
In the lesson you can change this and run it.
Each row of X is one day: [x, 1]. The product X @ [w, b] gives w·x + b for every row at once, lesson 55's matrix times a vector. The column of ones is what carries the intercept: it multiplies b by 1 in every row. np.linalg.lstsq finds the w and b with the smallest sum of squared residuals, 1.2 and 1.8 again, with residual total 2.8. With more features, X just gets more columns, and the same line of code fits a plane or more.
Quick check
What does this print?
import numpy as np
x = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
y = np.array([3.0, 5.0, 4.0, 7.0, 8.0])
pred = 1.2 * x + 1.8print((y - pred).round(2).tolist(), round(float(np.sum((y - pred) ** 2)), 2))
[0.0, 0.8, -1.4, 0.4, 0.2] 2.8
[0.0, -0.8, 1.4, -0.4, -0.2] 2.8
[3.0, 4.2, 5.4, 6.6, 7.8] 2.8
[0.0, 0.8, -1.4, 0.4, 0.2] 0.0
Show the answer
[0.0, 0.8, -1.4, 0.4, 0.2] 2.8
The residuals are y − ŷ; they add to 0, but their squares add to 2.8.
Drill
Fit, predict, report
Write fit(x, y) that returns the least-squares w and b by the formula. Then set w and b for the five days, pred6 to the prediction for day 6, and mse to the mean squared error on the five days, each rounded to 4 places.
import numpy as np
deffit(x, y):
w = 0.0
b = y.mean()
return w, b
x = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
y = np.array([3.0, 5.0, 4.0, 7.0, 8.0])
w, b = fit(x, y)
pred6 = 0
mse = 0print(w, b, pred6, mse)
In the lesson, your code is checked as you write it.
In your own words
Gradient descent needed 2,000 steps to reach a line the formula gives in one line. Why learn descent at all?
Write a sentence or two, then compare it with a good answer in the lesson.
In AI and ML: train, then test on days held back
Purpose. Use the line the way every model is used: fit on some data, judge on other data.
Twelve days of temperature and rentals. Eight are used for fitting: temperatures 14, 16, 18, 20, 22, 24, 26 and 28 with rentals 15, 19, 26, 30, 33, 41, 44 and 51. Four are held back: 17, 21, 25 and 29 degrees, with 24, 30, 41 and 50 rentals. Fit the line on the eight training days by the formula, then measure its mean squared error on each set:
The line is ŷ = 2.53t − 20.75: each extra degree brings about two and a half more tents. On the eight training days its mean squared error is 1.09; on the four held-back days it's 4.44. A higher error on new days is normal: the line was chosen to fit the training days, so it partly fits their noise, and the test days have noise of their own. With only four of them the number is rough, but it is the honest estimate of how the model will do tomorrow. A line has just two settings, so it can't chase the noise far. A model with many settings can push its training error toward 0 while its test error climbs: it has learned the particular days, not the pattern. That is overfitting, and the fix is a simpler model or more data.
Quick check
A model has training error 0.1 and test error 9.5. What does that suggest?
The test data must be wrong
It needs more training steps to bring the test error down
It overfits: it learned the noise, not the pattern
It's an excellent model
Show the answer
It overfits: it learned the noise, not the pattern
A test error far above the training error means the fit doesn't carry over to new data.
In PyTorch. In the real tool, a line is a layer with one input and one output, nn.Linear(1, 1): its weight is w and its bias is b. torch.optim.SGD is the descent loop from the Python part, written once for any model. Each pass computes the loss, clears the old slopes with zero_grad(), finds new ones with backward() and steps downhill with step() (lesson 79). Trained on the five days from the math part, the layer starts from random numbers:
import torch
import torch.nn as nn
x = torch.tensor([[1.0], [2.0], [3.0], [4.0], [5.0]])
y = torch.tensor([[3.0], [5.0], [4.0], [7.0], [8.0]])
model = nn.Linear(1, 1)
opt = torch.optim.SGD(model.parameters(), lr=0.05)
for step inrange(1000):
loss = ((model(x) - y) ** 2).mean()
opt.zero_grad()
loss.backward()
opt.step()
print(round(model.weight.item(), 4), round(model.bias.item(), 4), round(loss.item(), 4))
It ends on 1.2 and 1.8, the w and b the formula printed in the Python part's first cell, with a loss of 0.56, the math part's mean squared error. Every model PyTorch trains goes through this same loop. A classic model like this one doesn't need it: scikit-learn, the library for classic models, fits the line in one call.
Quick check
What does this print?
import numpy as np
w, b = 2.53, -20.75
t = np.array([17.0, 29.0])
print((w * t + b).round(2).tolist())
[43.01, 73.37]
[22.26, 73.37]
[22.26, 52.62]
[24.0, 50.0]
Show the answer
[22.26, 52.62]
The trained line predicts 2.53 × 17 − 20.75 = 22.26 tents at 17 degrees and 52.62 at 29. The days held back had 24 and 50.
The same judgment in matrix form. The Python part's worked example wrote the line as ŷ = Xw, with a column of ones in X to carry b. That product, inputs times weights plus a bias, is what every layer of a network computes, up to a transformer's last one. So judge the line that way too: one X for the training days, one for the held-back days, and one w for both.
Drill
Train error and test error, in matrix form
The starter fits wb on the training days in matrix form, but its X_test has the wrong second column: fix it. Then set train_mse and test_mse to the mean squared error of X @ wb on each set, and gap to test_mse − train_mse, each rounded to 2 places.
31Formulas with fractions: clear them firstFreeClearing denominators, cross-multiplying, a letter in the bottom, and reciprocal formulas like 1/R = 1/R₁ + 1/R₂.35 min
37Inverse functions, lines and scalingFreeUndoing a function by making x the subject, domains that allow it, line forms, and unscaling predictions.40 min
38Sequences and sums: find n, a, d and rFreeEvery letter of the arithmetic and geometric formulas as the subject, and an average's window from its decay.40 min
40Review: rearrange anything (algebra)FreeBroken rearrangements to fix, mixed practice from the whole algebra block, a strategy chart, and its formula sheet.35 min
41Coordinates and distanceFreePoints as (x, y), Pythagoras, distance in any number of coordinates, and nearest neighbors.42 min
42Angles, degrees and radiansFreeMeasuring a turn, converting units, arcs, and the angle between two arrows.36 min
43Sine and cosineFreeAcross and up: right triangles, the unit circle, and directions and rotations.42 min
44Waves: height, speed and shiftFreeAmplitude, period and phase, sampling a wave, and position codes in transformers.37 min
68Vector equations: rearranging with arrowsFreeSolving for a vector or a scale, recovering a gradient from two weight snapshots, and an angle from a dot product.40 min
87Counting: choices, orders and groupsFreeThe counting principle, factorials, ordered picks and groups, and why attention's cost grows with the square of the length.43 min
88Probability: from counts to chancesFreeEqually likely outcomes, the complement, distributions, seeded simulation, and a next-word model.43 min
89Combining events: and, or, independentFreeThe or rule, independence, probability trees, drawing with and without replacement, and chains of steps.42 min
100Maximum likelihoodFreeLikelihoods as products, log-likelihoods as sums, grid search, underflow, and the negative log-likelihood loss.45 min
101Linear regression: the best lineFreeResiduals, least squares by formula, three roads to one line, the matrix form, and train versus test error.45 min
109One neuron, start to finishFreeWeighted sum, activation, loss, slopes and update, by hand and in code, and what one straight line can and can't split.45 min
111Training a small networkFreeThe backward pass in matrix form, gradient checks, a full training loop, and reading loss curves.45 min
112The same network in PyTorchFreenn.Sequential with the NumPy weights loaded, the same loss and slopes, then SGD and mini-batches.45 min
113Attention as matrix mathFreeScaled dot-product scores, the causal mask, a row-by-row softmax, and blending value vectors.45 min
114Residual connections and layer normFreeAdding the input back and why its slope keeps learning alive, layer norm by hand, and add and norm.40 min
115A transformer block, start to finishFreeA sentence through embeddings, attention, add and norm, a small network and next-word scores, every shape named.45 min
117Final test: transpose any formulaFreeSlips to fix from every topic, mixed practice of every kind of rearranging, the strategy chart and the full sheet.45 min
Learn the math behind AI from arithmetic up: numbers, algebra, vectors and matrices, calculus, probability and statistics, each shown as numbers moving and checked in Python, ending with a network you build and train from the formulas.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write