Distributions

Four shapes worth knowing

A few distributions turn up again and again. Each one has an everyday picture.

Bernoulli. One yes-or-no try, like one coin toss. Did this visitor buy? The only setting is $p$, the chance of a "yes."

Binomial. Repeat the same yes-or-no try $n$ times and count the yeses. The tries must be independent: one doesn't change the others. Two coin tosses is a binomial with $n = 2$. So is "buyers among 200 visitors." The chance of exactly $k$ yeses is:

In plain words: count the ways to get k yeses, then multiply by the chance of any one of those ways. The part $\binom{n}{k}$, said "n choose k," is the number of ways to pick which k of the n tries were yeses.

Check it on two coin tosses, with one head ($n = 2$, $k = 1$, $p = 0.5$):

• ways to pick which toss was heads: 2 (the first or the second);
• chance of one particular pattern, such as heads-then-tails: 0.5 × 0.5 = 0.25;
• total: 2 × 0.25 = 0.5.

That matches the table. Python's math.comb does the counting:

import math

n, k, p = 2, 1, 0.5
print(math.comb(n, k))
print(math.comb(n, k) * p**k * (1 - p)**(n - k))

It prints 2 (the number of ways) and 0.5 (the chance).

Poisson. Counts of events in a fixed window of time, when they happen independently at a steady average rate. Think orders per hour or support tickets per day. Its one setting is the average rate, written $\lambda$ (lambda).

Categorical. One draw from several labeled outcomes, each with its own chance. Which of five products a visitor buys. Which word comes next. This is the distribution a language model outputs.

The code below fakes 10,000 days of a shop. Each day, 200 visitors each buy with a 4% chance (binomial). It also fakes 10,000 hours of orders arriving at 3.5 per hour on average (Poisson). Then it draws a bar chart of each and prints the averages.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(0)
conversions = rng.binomial(n=200, p=0.04, size=10_000)     # 10,000 days of 200 visitors each
orders_per_hour = rng.poisson(lam=3.5, size=10_000)

fig, axes = plt.subplots(1, 2, figsize=(8, 3))
axes[0].hist(conversions, bins=np.arange(0, 21) - 0.5)
axes[0].set_title("conversions from 200 visitors (p = 4%)")
axes[1].hist(orders_per_hour, bins=np.arange(0, 15) - 0.5)
axes[1].set_title("orders per hour (rate 3.5)")
print("average conversions:", conversions.mean().round(2), " average orders/hour:", orders_per_hour.mean().round(2))

The averages come out close to 8 and 3.5. The 8 makes sense: 4% of 200 visitors is 8. A conversion here means a visitor who buys.

Now look at the left chart. A typical day has about 8 conversions. Yet anywhere from about 3 to 13 is ordinary noise. Seeing 6 one day and 11 the next isn't news.

import math

n, p = 10, 0.2

def chance(k):
    return math.comb(n, k) * p**k * (1 - p)**(n - k)

print("exactly 2:", round(chance(2), 3))
print("at most 1:", round(chance(0) + chance(1), 3))
print("all chances add to:", round(sum(chance(k) for k in range(n + 1)), 3))

Sampling the next token

Here is where distributions meet AI. A language model gives every possible next token a score. A step called softmax turns those scores into chances that add up to 1 (a later lesson covers it). That is a categorical distribution. The program then draws one token from it.

Picture a raffle drum with 100 tickets. 42 say "great," 30 say "good," 15 say "solid," 8 say "fine" and 5 say "amazing." Draw a ticket, read it, and put it back. Do that 1,000 times and "great" comes up about 42% of the time. Any single draw, though, could be any word. That's why the same prompt can give different replies.

The code draws 1,000 tokens this way, prints each token's share, then shows ten draws in a row:

import numpy as np

tokens = ["great", "good", "solid", "fine", "amazing"]
probs = [0.42, 0.30, 0.15, 0.08, 0.05]

rng = np.random.default_rng(7)
draws = rng.choice(tokens, size=1000, p=probs)
for t in tokens:
    print(f"{t:<8} {np.mean(draws == t):.3f}")
print("ten samples:", " ".join(rng.choice(tokens, size=10, p=probs)))

Each share lands close to its chance, and the ten samples mix different words.

The other option is to always pick the single most likely token. This is called greedy decoding. Math books write the choice as $\arg\max$, short for "the option with the biggest value." Greedy decoding is more predictable, but it also repeats itself more. Phase 5 covers how temperature and other settings reshape the distribution before a token is drawn.

words = ["cat", "dog"]
probs = [1.0, 0.0]
import random
print(random.choices(words, probs)[0])