Index Ai Academy
Foundation · Foundation mathStatistics and information

Softmax: scores into probabilities

About 43 minLooking ahead: some prerequisites aren't done yet

Why this matters

Lesson 102 turned one score into the chance of a yes. Most questions have more than two answers: which of three sign types is this, which of 50,000 words comes next. A model produces one score per answer, any numbers at all, negative or huge, and needs a fair way to turn them into a distribution: every chance positive, all of them adding up to 1, and a bigger score always getting a bigger share. Softmax is that way, and it sits at the very end of nearly every classifier and every language model. Its two practical details matter as much as the formula: the same scores can be "cooled" or "heated" with a temperature, which is the setting a chatbot exposes to make its replies safer or more varied, and a naive implementation overflows on scores a real model produces every day.

  • Turn any list of scores into chances that add up to 1: exponentiate, then divide by the total
  • See that adding the same number to every score changes nothing, and that temperature sharpens or flattens the result
  • Compute softmax with a loop, then in NumPy, subtracting the largest score first so nothing overflows
  • Read softmax as the last step of every classifier and language model, pick the predicted class with argmax, and run it in PyTorch with F.softmax, torch.logsumexp and F.log_softmax
Quick check
Have a guess before you start. Three scores, 2, 1 and 0.1, must become three chances adding up to 1. Why not just divide each by their total, 3.1?

The math: exponentiate, then share

Purpose. A fair way to turn any list of scores into chances that add up to 1.

Two steps. First make every score positive with the exponential: eᶻ is always above 0, and bigger z gives bigger eᶻ (lesson 24). Then divide each by their total, as lesson 88 turned counts into shares. A short table of e-values does the hand work:

z−100.1123
eᶻ0.36811.1052.7187.38920.09

For the scores 2, 1 and 0.1: e² + e¹ + e⁰·¹ = 7.389 + 2.718 + 1.105 = 11.21. The shares are 7.389 ÷ 11.21 = 0.659, 2.718 ÷ 11.21 = 0.242 and 1.105 ÷ 11.21 = 0.099. They're all positive, they add up to 1, and the order of the scores is kept.