Index Ai Academy
Foundation · Foundation mathStatistics and information

Linear regression: the best line

About 45 minLooking ahead: some prerequisites aren't done yet

Why this matters

Linear regression is the simplest model that is trained, tested and used exactly the way large models are. It predicts a number from a number, tents from temperature, with a straight line, and "training" means choosing the line's slope and intercept from data. Nearly every idea of the last four modules meets here: the line from lesson 21, covariance and variance from lessons 94 and 97, gradient descent from lesson 79, solving equations from lesson 60, and the matrix product from lesson 56. They all arrive at the same line, which makes this lesson a good place to see that they're one subject. And the most important habit in machine learning starts here too: judging a model on data it has never seen, not on the data it was trained on, because the error on training data is almost always too flattering.

  • Measure how well a line fits with residuals and the sum of their squares
  • Find the least-squares line by formula: w = cov(x, y)/var(x) and b = ȳ − w·x̄
  • Get the same line three ways in Python: the formula, gradient descent, and solving a small system
  • Fit on some days and judge on days held back, in the matrix form ŷ = Xw, and train the same line as a PyTorch layer with nn.Linear and optim.SGD
Quick check
Have a guess before you start. Five points don't lie on one line. What would "the best line" through them make as small as possible?

The math: the line that misses the least

Purpose. Choose the slope and intercept that fit the points best.

Residuals. A line ŷ = wx + b predicts ŷ ("y hat") for each x. A point's residual is how far the line misses it: y − ŷ. Five days, with x the day and y the rentals: (1, 3), (2, 5), (3, 4), (4, 7), (5, 8). The line ŷ = 1.2x + 1.8 predicts 3, 4.2, 5.4, 6.6 and 7.8, so the residuals are 0, 0.8, −1.4, 0.4 and 0.2.

Squared residuals. Some residuals are positive and some negative, so, as with variance in lesson 97, they're squared before adding: 0 + 0.64 + 1.96 + 0.16 + 0.04 = 2.8. That total is the sum of squared residuals, and divided by the number of points it's the mean squared error, 0.56, lesson 79's loss. The least-squares line is the one with the smallest total, and 1.2x + 1.8 is it: no other line through these points gets below 2.8.