Index Ai Academy
Foundation · Foundation mathStatistics and information

Maximum likelihood

About 45 minLooking ahead: some prerequisites aren't done yet

Why this matters

So far the chances have been given: a fair coin, a die, a weighted spinner. In practice it runs the other way round. You have the data, ten tosses or a million sentences, and you need the setting that produced it: the coin's p, or the billions of numbers inside a language model. Maximum likelihood is the rule almost every model is trained by: choose the setting under which the data you actually saw is least surprising. For a coin it gives the obvious answer, 7 heads in 10 means p = 0.7, but the same rule, applied to a network's predictions, is exactly what a training loop does. The loss a classifier or a language model minimizes, the negative log-likelihood, is this lesson's quantity with a minus sign. And the log is not decoration: without it, the probability of a long text rounds to 0 in any computer.

  • Compute the likelihood of some data for a given setting: the product of each observation's probability
  • Take its log to turn the product into a sum, and find the setting that makes the data most likely
  • Search a grid of settings with np.argmax, plot the log-likelihood, and see why a product of many chances underflows while a sum of logs doesn't
  • Recognize training as maximum likelihood: minimizing the negative log-likelihood, the loss classifiers and language models use, in NumPy and with PyTorch's F.nll_loss
Quick check
Have a guess before you start. A coin lands heads 7 times in 10 tosses. Which value of p, the chance of heads, best explains that?

The math: the setting that makes the data least surprising

Purpose. Choose a model's setting from the data it should explain.

Likelihood. For a coin with chance p of heads, the record H H T has probability p × p × (1 − p): the tosses are independent, so their chances multiply (lesson 89). Read as a function of p, with the data fixed, this is the likelihood L(p). For the record of 10 tosses with 7 heads and 3 tails:

pL(p) = p⁷(1 − p)³
0.50.000977
0.70.002224
0.90.000478

All three are small, because any one exact sequence of 10 tosses is unlikely. What matters is the comparison: p = 0.7 makes this record more than twice as likely as a fair coin does. Maximum likelihood picks the p with the largest L(p).