Index Ai Academy
Foundation · Foundation mathStatistics and information

Logistic regression: a line that answers yes or no

About 45 minLooking ahead: some prerequisites aren't done yet

Why this matters

Linear regression predicts an amount. Many questions want a yes or no instead: is this email spam, is this sign an arrow, will this hiker need a rescue. A model for those should answer with a chance between 0 and 1, and a straight line can't: it runs off below 0 and above 1. Logistic regression keeps the line as a score and squeezes the score through the sigmoid, an S-shaped curve that turns any number into a chance. It is trained by maximum likelihood (lesson 100), and its loss, the log loss, has a slope so simple, p − y, that the same expression returns in nearly every classifier and language model. A single neuron with a sigmoid is exactly this model, so this lesson is also the first complete neuron of the section: score, squeeze, loss, slope, step.

  • Turn a line's score into a chance with the sigmoid, σ(z) = 1/(1 + e⁻ᶻ), and read it off the S-curve
  • Measure a yes-or-no prediction with the log loss, and see why it punishes confident mistakes
  • Derive and use the loss's slope with respect to the score, p − y, in a short training loop
  • Train a two-feature sign detector, read it as one neuron, choose a threshold, and get the same loss and slopes from PyTorch's nn.BCEWithLogitsLoss
Quick check
Have a guess before you start. A model should give the chance a sign is an arrow. Why not use a straight line, ŷ = wx + b, as the chance?

The math: score, squeeze, loss

Purpose. Predict the chance of a yes, and measure how good that prediction was.

The score. As in linear regression, each example gets a score z = w·x + b: a weighted sum of its features plus a bias. A high score means "probably yes", a low one "probably no", but it can be any number.

The squeeze. The sigmoid turns the score into a chance:

z−4−2024
σ(z)0.0180.1190.50.8810.982

A score of 0 gives 0.5, a coin toss. Big positive scores approach 1 and big negative ones approach 0, never reaching either. The decision boundary is where z = 0: on one side the model leans yes, on the other no.