Index Ai Academy
Phase 3 · Reinforcement learning

Policy gradients: learning what to do directly

A policy as probabilities you can nudge: turn up what worked, measure it against a baseline, and see how the same update trains a language model.

About 40 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

Q-learning learns how good each action is and then picks the best. There is a more direct way: keep the policy itself as a set of adjustable numbers, try things, and turn up the probability of whatever worked. That is the method behind the reinforcement learning that trains language models on human preferences (RLHF) and on problems with checkable answers (Training reasoning models). A language model already is a policy: it gives a probability to every next word. This lesson builds the update on five headlines and a small grid, and the last part carries it across to models.

What you'll be able to do

  • Represent a policy as probabilities from a softmax over preferences, and sample actions from it
  • Apply the policy gradient update: raise the probability of an action in proportion to the return that followed it
  • Explain why rewards are measured against a baseline, and show what goes wrong without one
  • Say how the same update trains a language model: the model is the policy and each word is an action
Quick check
A guess before you start. A learner gives each of five headlines a probability, shows one, and after every reward raises the probability of the headline it just showed in proportion to the reward. Now suppose every reward is positive: 10 for no click and 11 for a click. What will the learner do?
  1. Find the best headline as before, since a click still pays more
  2. Lock on to whichever headline it happens to show first, good or bad
  3. Keep all five probabilities equal, since every reward is positive
  4. Find the best headline faster, since larger rewards mean larger steps
Show the answer

Lock on to whichever headline it happens to show first, good or bad

If every outcome is rewarded, every choice is reinforced. Whatever the learner shows first gets more likely, so it is shown again, and reinforced again. The difference between 10 and 11 is real but tiny beside the 10 that everything earns. The fix, measuring each reward against what is normal, is the third part of this lesson.

The idea: preferences, probabilities and a nudge

Purpose. Keep a policy as numbers you can adjust, and raise the probability of whatever worked.

A Q-learning agent's policy is implied by its table: take the largest value. A policy gradient method stores the policy itself. For each action it keeps a number called a preference, and the softmax function from the softmax lesson turns the preferences into probabilities. The agent then samples its action from those probabilities.

Setup (runs before every cell) · 29 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import numpy as np

true_rates = np.array([0.20, 0.50, 0.35, 0.65, 0.45])


def softmax_rows(x):
    e = np.exp(x - x.max(axis=1, keepdims=True))
    return e / e.sum(axis=1, keepdims=True)


def reinforce(baseline, bonus=0.0, alpha=0.3, steps=2000, players=500, seed=0):
    """500 players with five preferences each: clicks per step, and the probability
    each gives the best headline."""
    rng = np.random.default_rng(seed)
    arms, rows = len(true_rates), np.arange(players)
    prefs, average = np.zeros((players, arms)), np.zeros(players)
    clicks, best = np.zeros(steps), np.zeros(steps)
    for t in range(steps):
        probs = softmax_rows(prefs)
        choice = (rng.random((players, 1)) > probs.cumsum(axis=1)).sum(axis=1)
        click = (rng.random(players) < true_rates[choice]).astype(float)
        reward = click + bonus
        average += (reward - average) / (t + 1)
        signal = reward - average if baseline else reward
        direction = -probs
        direction[rows, choice] += 1.0
        prefs += alpha * signal[:, None] * direction
        clicks[t], best[t] = click.mean(), probs[:, true_rates.argmax()].mean()
    return clicks, best
import numpy as np

def softmax(x):
    e = np.exp(x - x.max())
    return e / e.sum()

prefs = np.array([0.0, 1.0, 2.0])                 # one preference per action
probs = softmax(prefs)
print("probabilities:", probs.round(3))

rng = np.random.default_rng(0)
draws = rng.choice(3, size=1000, p=probs)         # sample an action 1,000 times
print("chosen in 1,000 draws:", np.bincount(draws, minlength=3))
It runs right here in your browser, with nothing to install.

The third action has the highest preference, so it gets a probability of 0.665 and is chosen 697 times in a thousand. The first is still chosen 84 times. A policy like this explores on its own: nothing is ever ruled out, and how often something is tried is set by its probability.

Learning will mean changing the preferences. That has three advantages worth knowing.

  • It is smooth. A small change to a preference is a small change in behavior. A Q-table's policy can flip completely when one value overtakes another.
  • It scales to huge action sets. A language model has tens of thousands of possible next words. It can't keep a separate table row for each, but it already outputs a probability for every one.
  • The policy can be random on purpose. In rock-paper-scissors the best policy is to play each a third of the time. A probability can say that. "Take the largest Q-value" can't.
Quick check
From the softmax lesson: three actions have preferences 2.0, 2.0 and 2.0. What probabilities does the softmax give them, and what happens if 5 is added to all three?
  1. 0.2 each, since each preference is 2.0 out of a total of 6.0
  2. 1.0 each, since they are tied for the largest preference
  3. A third each, and adding 5 to all of them changes nothing
  4. A third each, and adding 5 makes each of them more likely
Show the answer

A third each, and adding 5 to all of them changes nothing

Softmax turns equal scores into equal probabilities: a third each. And it depends only on the differences between the scores, so adding the same number to every preference leaves every probability as it was.

Turn up what worked

Here is the update. The agent samples an action aa, and some return GG follows. It then changes every preference:

θb←θb+α G ([b=a]−πb)\theta_b \leftarrow \theta_b + \alpha \, G \, \big( [b = a] - \pi_b \big)

In plain words: θb\theta_b (theta) is the preference for action bb and πb\pi_b (pi) is its current probability. [b=a][b = a] is 1 for the action that was taken and 0 for the others. So the action taken moves by αG(1−πa)\alpha G (1 - \pi_a), up if the return was positive. Every other action moves by −αGπb-\alpha G \pi_b, the opposite way. α\alpha is the step size.

A good outcome makes the chosen action more likely and the rest less. A bad one, a negative GG, does the reverse. And the size of the push scales with the size of the return.

Uses the setup above, 29 lines.

import numpy as np

def softmax(x):
    e = np.exp(x - x.max())
    return e / e.sum()

alpha = 0.5
prefs = np.array([0.0, 0.0, 0.0])
probs = softmax(prefs)
action = 1                                        # the action that was sampled

direction = -probs
# 1 for the action taken, minus every probability
direction[action] += 1.0
print("probabilities before:", probs.round(3))
print("direction:           ", direction.round(3))
print("after a return of +1:", softmax(prefs + alpha * 1.0 * direction).round(3))
print("after a return of -1:", softmax(prefs + alpha * -1.0 * direction).round(3))

From three equal probabilities, a return of +1 after the middle action takes it from 0.333 to 0.452. A return of −1 takes it down to 0.233 and shares the difference between the other two.

Where does that direction come from? It is a gradient, in the sense of the gradient descent lesson: the slope of the log of the chosen action's probability. Multiplying it by the return and stepping along it, over many samples, climbs toward policies with a higher expected return. That is why these are called policy gradient methods. This simplest one has a name of its own, REINFORCE.

Now the five headlines from the bandit lesson, with a softmax policy and this update:

Uses the setup above, 29 lines.

import numpy as np

true_rates = np.array([0.20, 0.50, 0.35, 0.65, 0.45])

def softmax_rows(x):
    e = np.exp(x - x.max(axis=1, keepdims=True))
    return e / e.sum(axis=1, keepdims=True)

def reinforce(baseline, bonus=0.0, alpha=0.3, steps=2000, players=500, seed=0):
    """500 players, each with five preferences. Returns clicks per step and the
    probability each gives the best headline."""
    rng = np.random.default_rng(seed)
    arms, rows = len(true_rates), np.arange(players)
    prefs, average = np.zeros((players, arms)), np.zeros(players)
    clicks, best = np.zeros(steps), np.zeros(steps)
    for t in range(steps):
        probs = softmax_rows(prefs)
        # sample one headline per player
        choice = (rng.random((players, 1)) > probs.cumsum(axis=1)).sum(axis=1)
        click = (rng.random(players) < true_rates[choice]).astype(float)
        reward = click + bonus
        # each player's mean reward so far
        average += (reward - average) / (t + 1)
        signal = reward - average if baseline else reward
        direction = -probs
        # 1 for the arm shown, minus the probabilities
        direction[rows, choice] += 1.0
        prefs += alpha * signal[:, None] * direction
        clicks[t], best[t] = click.mean(), probs[:, true_rates.argmax()].mean()
    return clicks, best

clicks, best = reinforce(baseline=False)
print("probability of the best headline at steps 1, 100, 500, 2000:",
      [round(float(best[t - 1]), 2) for t in (1, 100, 500, 2000)])
print("clicks per 1000, last 1000 visitors:", round(float(clicks[-1000:].sum())))

The players start with a fifth of their probability on the best headline. After 2,000 visitors they give it 0.95, and collect 639 clicks per thousand, close to the 650 that is the most possible. No table of values was kept, and no rule about exploring was written: the probabilities did both jobs.

(Ignore the baseline and bonus arguments for a moment. They are for the next part.)

Quick check
Three actions have probabilities 0.5, 0.3 and 0.2. The agent takes the first and a return of +2 follows. With a step size of 0.1, what does this print: the change to each preference?
import numpy as np

probs = np.array([0.5, 0.3, 0.2])
action, G, alpha = 0, 2.0, 0.1
direction = -probs
direction[action] += 1.0
print((alpha * G * direction).round(2))
  1. [ 0.1 0.06 0.04]
  2. [ 0.1 -0.06 -0.04]
  3. [ 0.2 -0.06 -0.04]
  4. [ 0.5 -0.3 -0.2 ]
Show the answer

[ 0.1 -0.06 -0.04]

The direction is 1 − 0.5 = 0.5 for the action taken and −0.3 and −0.2 for the others. Multiplied by α × G = 0.2, the changes are +0.1, −0.06 and −0.04. They add up to zero: probability moved toward the action that worked, away from the rest.

Quick check
From cross-entropy: the gradient of the loss −log π_a with respect to the scores is π − y, where y is 1 for the action a and 0 for the others. What does the policy gradient direction [b = a] − π equal?
  1. The loss itself, a single number
  2. The same slope, π − y, with the sign unchanged
  3. The probability π of the action taken
  4. The same slope with its sign turned: y − π
Show the answer

The same slope with its sign turned: y − π

Descending the cross-entropy loss −log π_a moves the scores along −(π − y) = y − π, which is [b = a] − π. So the policy gradient direction is exactly the slope of the log probability of the action taken. Climbing it, scaled by the return, raises the probability of actions that worked.

Drill

One policy gradient update

Write policy_update(prefs, probs, action, signal, alpha). prefs and probs are lists of the same length, action is the index of the action that was taken, and signal is the return (or advantage) that followed. Return a new list of preferences: the action taken moves by alpha * signal * (1 - probs[action]), and every other action b by -alpha * signal * probs[b]. Don't change the lists you were given.

Uses the setup above, 29 lines.

def policy_update(prefs, probs, action, signal, alpha):
    return list(prefs)


print(policy_update([0.0, 0.0, 0.0], [1 / 3, 1 / 3, 1 / 3], 1, 1.0, 0.5))
In the lesson, your code is checked as you write it.

The picture. One update, on three equal preferences. The middle action was sampled. After a return of +1 its probability rises from 0.333 to 0.452 and the other two share the loss. After a return of −1 the same push goes the other way. The numbers are the ones the cell above prints.

Try it now

The three probabilities always add to 1: what one action gains, the others lose.

In your own words
Describe what one policy gradient update does to the probabilities after a good outcome, and after a bad one.

Write a sentence or two, then compare it with a good answer in the lesson.

In code: a baseline, many steps and PyTorch

Purpose. Fix what goes wrong when every reward is positive, then run the same update over a whole episode and in PyTorch.

Better than what?

The update pushes toward an action whenever the return is positive. That hides a trap, the one from the opening question. Take the same five headlines and add 10 to every reward: 10 for no click, 11 for a click. Nothing about the problem has changed. The best headline is still the best, by exactly as much.

Uses the setup above, 29 lines.

import numpy as np

true_rates = np.array([0.20, 0.50, 0.35, 0.65, 0.45])

def softmax_rows(x):
    e = np.exp(x - x.max(axis=1, keepdims=True))
    return e / e.sum(axis=1, keepdims=True)

def reinforce(baseline, bonus=0.0, alpha=0.3, steps=2000, players=500, seed=0):
    rng = np.random.default_rng(seed)
    arms, rows = len(true_rates), np.arange(players)
    prefs, average = np.zeros((players, arms)), np.zeros(players)
    clicks, best = np.zeros(steps), np.zeros(steps)
    for t in range(steps):
        probs = softmax_rows(prefs)
        choice = (rng.random((players, 1)) > probs.cumsum(axis=1)).sum(axis=1)
        click = (rng.random(players) < true_rates[choice]).astype(float)
        reward = click + bonus
        # each player's mean reward so far
        average += (reward - average) / (t + 1)
        signal = reward - average if baseline else reward
        direction = -probs
        direction[rows, choice] += 1.0
        prefs += alpha * signal[:, None] * direction
        clicks[t], best[t] = click.mean(), probs[:, true_rates.argmax()].mean()
    return clicks, best

for name, baseline, bonus in (("no baseline, rewards 0 or 1  ", False, 0.0),
                              ("no baseline, rewards 10 or 11", False, 10.0),
                              ("baseline,    rewards 0 or 1  ", True, 0.0),
                              ("baseline,    rewards 10 or 11", True, 10.0)):
    clicks, best = reinforce(baseline, bonus)
    print(name, "| clicks per 1000, last 1000 visitors:",
          round(float(clicks[-1000:].sum())),
          "| probability of the best headline at the end:", round(float(best[-1]), 2))

Look at the second row. With rewards of 10 and 11 the learner collapses: 444 clicks, about what choosing at random gives, and only 0.23 of its probability on the best headline. Every choice earned at least 10, so every choice was reinforced hard, and each player locked on to whatever it tried first.

The last two rows fix it with one subtraction. In place of the reward, use the reward minus a baseline: a measure of what is normal. Here the baseline is the player's own average reward so far, the latest one included. The signal is now "better or worse than usual": positive for a click, negative for none, whatever the rewards' scale. With it, the learner gets 648 clicks in both versions of the problem, to the last digit.

The four runs as a picture. The probability each player gives the best headline, over the 2,000 visitors. The setup holds the reinforce function from the cells above:

Uses the setup above, 29 lines.

import matplotlib.pyplot as plt

runs = [("no baseline, rewards 0 or 1", False, 0.0),
        ("no baseline, rewards 10 or 11", False, 10.0),
        ("baseline, rewards 0 or 1", True, 0.0),
        ("baseline, rewards 10 or 11", True, 10.0)]
for label, baseline, bonus in runs:
    clicks, best = reinforce(baseline, bonus)
    plt.plot(best, label=label)
plt.xlabel("visitor")
plt.ylabel("probability of the best headline")
plt.legend()
plt.show()

The two baseline curves lie on top of each other: shifting every reward by 10 changes nothing once the baseline is subtracted. The no-baseline curve with rewards 10 or 11 stays near the bottom.

Quick check
A team trains a policy with rewards between 0 and 1 and it learns well. They then change the scoring so that rewards run between 100 and 101, and training stops working. What is the most likely cause?
  1. The rewards are too large for the softmax to turn into probabilities
  2. No baseline: every action now earns a large positive reward and is reinforced
  3. The best action has changed now that the scores are higher
  4. The step size should be raised to match the larger rewards
Show the answer

No baseline: every action now earns a large positive reward and is reinforced

Shifting every reward by 100 doesn't change which action is best. But an update that multiplies by the raw reward now pushes hard toward whatever was sampled, and the difference of 1 is lost. Subtracting a baseline, such as the average reward, restores the signal.

The difference between what happened and what was expected has a name: the advantage.

A=G−bA = G - b

In plain words: AA is the advantage, GG the return that followed the action, and bb the baseline, the return you'd normally expect. An advantage above zero means the action did better than usual and should become more likely. Below zero, less likely.

Even when rewards are well scaled, a baseline helps. Compare the first and third rows: 639 clicks without, 648 with. Subtracting the expected part removes noise from every update without changing where the updates lead on average, so the learner steps more surely.

This is the same error you have seen in every lesson of this track: what happened, minus what was expected.

Quick check
An agent's baseline, its average return so far, is 10.4. It takes an action and the return is 10. What does this print: the advantage, and what happens to that action's probability?
G, baseline = 10.0, 10.4
advantage = G - baseline
print(round(advantage, 1), "more likely" if advantage > 0 else "less likely")
  1. 10.0 more likely
  2. -0.4 less likely
  3. 0.4 more likely
  4. -0.4 more likely
Show the answer

-0.4 less likely

The return of 10 is below the usual 10.4, so the advantage is −0.4 and the update makes the action less likely. Without the baseline the raw return, +10, would have made it much more likely, which is how the learner in the second row went wrong.

Drill

Advantages from a list of returns

Write advantages(returns). Go through the list in order. For each return, the baseline is the mean of the returns so far, this one included, as in the lesson's code, and the advantage is the return minus that baseline. So the first advantage is always 0: with nothing to compare it to, the first return counts as exactly usual. Return the list of advantages, each rounded to 3 places.

Uses the setup above, 29 lines.

def advantages(returns):
    return list(returns)


print(advantages([10, 11, 10, 11, 11]))
In the lesson, your code is checked as you write it.

Many steps, and the road to language models

So far each action had its own immediate reward. In a task with many steps, one change is needed: the signal for an action is the return from that step onward, since an action can't be credited for rewards that came before it. Here is the grid world again, learned by policy gradient with a baseline for each square.

Uses the setup above, 29 lines.

import numpy as np

ROWS, COLS, WALL, START = 3, 4, (1, 1), (2, 0)
ENDS = {(0, 3): 1.0, (1, 3): -1.0}
MOVES = {"up": (-1, 0), "down": (1, 0), "left": (0, -1), "right": (0, 1)}
ACTIONS = list(MOVES)
STATES = [(r, c) for r in range(ROWS) for c in range(COLS) if (r, c) != WALL]

def step(state, action):
    r, c = state[0] + MOVES[action][0], state[1] + MOVES[action][1]
    nxt = (r, c) if 0 <= r < ROWS and 0 <= c < COLS and (r, c) != WALL else state
    return nxt, ENDS.get(nxt, -0.04)

def softmax(x):
    e = np.exp(x - x.max())
    return e / e.sum()

def reinforce(episodes, alpha=0.2, gamma=0.9, seed=0):
    rng = np.random.default_rng(seed)
    prefs = {s: np.zeros(4) for s in STATES}          # four preferences in every square
    # the average return seen from each square: the baseline
    value = {s: 0.0 for s in STATES}
    seen = {s: 0 for s in STATES}
    returns = []
    for episode in range(episodes):
        s, path = START, []
        # play one episode by sampling from the policy
        while s not in ENDS and len(path) < 60:
            a = rng.choice(4, p=softmax(prefs[s]))
            nxt, reward = step(s, ACTIONS[a])
            path.append((s, a, reward))
            s = nxt
        G = 0.0
        # work backward: G is the return from each step onward
        for s, a, reward in reversed(path):
            G = reward + gamma * G
            advantage = G - value[s]
            seen[s] += 1
            value[s] += (G - value[s]) / seen[s]
            direction = -softmax(prefs[s])
            direction[a] += 1.0
            prefs[s] += alpha * advantage * direction
        returns.append(sum(reward for s, a, reward in path))
    return prefs, returns

prefs, returns = reinforce(1000)
print("mean return by hundred episodes:",
      [round(float(np.mean(returns[i:i + 100])), 2) for i in range(0, 1000, 100)])
print("policy at the start square:",
      dict(zip(ACTIONS, softmax(prefs[START]).round(2).tolist())))

The first hundred episodes average 0.30 and the second hundred 0.79. By the end the agent averages 0.83, beside a best possible 0.84, and at the start square it goes up with probability 0.97. It found the route Q-learning found, by a different road: no table of action values, only probabilities, pushed toward whatever turned out better than the square's average.

Quick check
A three-move episode earns the rewards 0, 0 and 1. With gamma 0.9, what does this print: the return from each move onward?
rewards, gamma = [0, 0, 1], 0.9
G, out = 0.0, []
for reward in reversed(rewards):
    G = reward + gamma * G
    out.append(round(G, 2))
print(out[::-1])
  1. [0.81, 0.9, 1.0]
  2. [1.0, 1.0, 1.0]
  3. [0.0, 0.0, 1.0]
  4. [1.0, 0.9, 0.81]
Show the answer

[0.81, 0.9, 1.0]

Working backward: the last move's return is 1.0. The move before it gets 0 + 0.9 × 1.0 = 0.9, and the first gets 0 + 0.9 × 0.9 = 0.81. Each move is credited with what followed it, discounted by how long it took to arrive.

The same update in PyTorch. In a library, nobody writes the direction by hand. The policy gives a log probability for the action taken, and the loss is that log probability times the advantage, with a minus sign so that a descent step climbs the probability. Backpropagation then produces the direction. The gradient on the preferences is the negative of the direction from the cell above:

Uses the setup above, 29 lines.

import torch
from torch.distributions import Categorical

prefs = torch.zeros(3, requires_grad=True)
dist = Categorical(logits=prefs)
action, advantage = torch.tensor(1), 1.0
loss = -dist.log_prob(action) * advantage
loss.backward()
print([round(g, 3) for g in (-prefs.grad).tolist()])

The printed list is the direction: +0.667 for the action taken and −0.333 for each of the others, exactly the numbers behind the first picture, with α = 1. The browser runs a teaching subset of PyTorch, the same code runs on a real install, and seeded random numbers differ between them, so only the structure is compared here.

Worked example

Worked example: one episode's update, by hand

Follow a single short episode on a corridor of two squares, each with two actions, and compute every number of the update: the returns from each step, the advantages against the squares' baselines, and the new probabilities.

Uses the setup above, 29 lines.

import numpy as np

def softmax(x):
    e = np.exp(x - x.max())
    return e / e.sum()

alpha, gamma = 0.5, 0.9
# actions: 0 = left, 1 = right
prefs = {"first": np.array([0.0, 0.0]), "second": np.array([0.0, 0.0])}
# the average return seen from each square so far
baseline = {"first": 0.2, "second": 0.5}

# went right (reward 0), then right again (reward 1, the end)
path = [("first", 1, 0.0), ("second", 1, 1.0)]

G = 0.0
for square, action, reward in reversed(path):
    G = reward + gamma * G
    advantage = G - baseline[square]
    direction = -softmax(prefs[square])
    direction[action] += 1.0
    prefs[square] = prefs[square] + alpha * advantage * direction
    print(square, "| return from here", round(G, 2), "| advantage", round(advantage, 2),
          "| probability of right is now", round(float(softmax(prefs[square])[1]), 3))
In the lesson you can change this and run it.

The loop runs backward, so the second square comes first. Its return is the final reward, 1.0, half a point above that square's baseline of 0.5, and the probability of going right there rises from 0.5 to 0.562. Then the first square: its return is 0 + 0.9 × 1.0 = 0.9. That move's own reward was 0, but it is credited with what it led to, and 0.9 is well above its baseline of 0.2, so its probability of going right rises more, to 0.587. One episode moved both squares a little. Thousands of episodes, each a small push toward what did better than expected, are the whole of the method.

In your own words
What is the advantage, and why is it a better signal than the raw return?

Write a sentence or two, then compare it with a good answer in the lesson.

In practice: the same update on a language model

Purpose. Map the lesson onto a language model, and see the one addition that real training methods make.

The same update, on a language model

Everything is now in place to see how a language model is trained with rewards, the subject of several lessons in the phase on modern LLM engineering.

In this lessonIn a language model
The stateThe text so far
An actionThe next word (token)
The policy's preferencesThe model's output scores, set by its weights
An episodeOne whole answer
The rewardA score for the answer: from people's ratings, a model trained to predict them, or a checker
The updateShift the weights so that the words of better-than-expected answers become more likely

The model writes an answer by sampling one word after another: an episode. The answer gets a score. The gradient that makes the sampled words more probable is computed by backpropagation (Backpropagation), in place of this lesson's one-line direction. It is multiplied by the advantage, and the weights take a small step. Methods such as PPO (RLHF) are this update with one main addition: a limit on how far the policy may move in a single step, because a policy that changes too fast collects its next batch of experience from somewhere it has never been.

The one addition real methods make

In your own words
Explain to a colleague how a language model can be improved with rewards, using the words policy, action, reward and advantage.

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 8 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 11 review cards for spaced repetition

Try a quiz question

What does a policy gradient method adjust as it learns?

  1. The preferences that set the probability of each action
  2. A table of the expected return of each action in each state
  3. The rewards the environment gives for each action
  4. The discount factor applied to later rewards

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 3 · about 39 hours of study

Machine learning

Learning from data, and knowing when it worked 50 lessons 1 free
  1. What machine learning is FreePrograms that learn rules from examples instead of being told them.30 min
  2. Features and labelsSupervised learning: inputs, targets and predictions.30 min
  3. Train, validation and test setsHold data back so you can trust your results.38 min
  4. Preparing dataClean, encode and scale data before a model sees it.40 min
  5. Linear regressionFit a line, and predict numbers from features.42 min
  6. Loss functionsOne number that says how wrong the model is.35 min
  7. Training with gradient descentAdjust weights step by step to reduce the loss.42 min
  8. Learning rateStep size, convergence and divergence.35 min
  9. Logistic regressionPredict probabilities for yes/no outcomes.42 min
  10. Multiclass classificationSoftmax and cross-entropy for more than two classes.42 min
  11. The same three models in PyTorchA line, logistic regression and softmax regression fitted both ways: the loss, the gradient and the update opened up, with weight decay and mini-batches.35 min
  12. Precision, recall and F1Accuracy lies; these don't.42 min
  13. ROC curves and AUCJudge a classifier across every threshold.30 min
  14. Calibration: can the probability be trusted?Bands of predicted risk against what happened, the Brier score, why naive Bayes is over-confident, and fixing it on rows the model did not fit.30 min
  15. Overfitting and underfittingMemorizing versus learning.30 min
  16. RegularizationPenalize complexity so models generalize.40 min
  17. Bias and varianceThe two ways a model can be wrong.30 min
  18. Cross-validationReuse limited data for more reliable estimates.42 min
  19. Imbalanced classes in depthRare classes measured: why rebalancing mostly moves the threshold, correcting inflated probabilities, average precision against AUC, why the count of rare cases matters, and thresholds from costs.45 min
  20. Naive Bayes: classifying with countsA classifier trained by counting: Bayes' rule over words, the naive assumption, log scores, smoothing for unseen words, why it is overconfident, and when it beats logistic regression.45 min
  21. Nearest neighborsPredict from the most similar examples.35 min
  22. Decision treesLearn a flowchart of yes/no questions.42 min
  23. Random forests and boostingMany weak models voting beat one strong one.42 min
  24. Gradient boosting in depthBoosting built from scratch: trees fit to the gradient of the loss, depth and interactions, learning rate against number of trees, early stopping, and what XGBoost and LightGBM add.45 min
  25. Support vector machinesThe widest margin, support vectors, the hinge loss and C, kernels for curved boundaries, and when an SVM fits.40 min
  26. Recommender systemsShrunk averages, item-to-item collaborative filtering, matrix factorization, and cold start.40 min
  27. Project: a price modelMultiple regression on log prices, one-hot categories, percentage effects and an extrapolation flag.45 min
  28. Feature engineeringTurn raw data into signals a model can use.35 min
  29. The scikit-learn workflowFit, predict and evaluate with the standard ML library.45 min
  30. Explaining a model: what it uses, and why this predictionPermutation importance on held-back rows, a linear model's contributions for one customer, partial dependence, and what none of them can say.35 min
  31. Capstone: churn predictionPredict which customers will leave, and measure it properly.45 min
  32. ClusteringGroup similar items without labels (k-means).35 min
  33. Clusters of any shape: DBSCAN and hierarchiesWhy k-means needs round clusters, DBSCAN's core points and noise, and hierarchical clustering with its linkages.40 min
  34. Anomaly detectionMasking, robust scores with the median and MAD, baselines from the past, isolation forests and thresholds from a review budget.40 min
  35. Dimensionality reductionVisualize and compress high-dimensional data.35 min
  36. Cleaning messy dataDuplicates, junk rows, text numbers, mixed dates and missing values that mean something.45 min
  37. Grouping, pivoting and reshapingNamed aggregation, pivot tables, transform, running totals and groups too small to rank.45 min
  38. Data in pieces: partitions and mergeable summariesSplit, summarize and combine; which statistics merge exactly, merging variances, sketches for the rest, and partitioning well.30 min
  39. Combining tables with mergeJoin on keys safely, stack tables, and divide by the right denominator.44 min
  40. Time series with pandasResample, smooth, compare across years, and test whether one series leads another.45 min
  41. Forecasting: baselines, lag features and backtestsThree baselines, lag features that never see their own future, a walk-forward backtest, and a range around the number.40 min
  42. Text mining: TF-IDF, search and labelsBag of words and TF-IDF, similarity search and its blind spots, text classifiers and topics, and when LLM labels are worth it.35 min
  43. Location dataHaversine distances, nearest places, GeoJSON, point in polygon and simple maps.40 min
  44. Comparing groups: t-testsWelch's t-test, pooled vs average rates, and a control group for before-and-after changes.40 min
  45. More tests: groups, categories, relationshipsChoose a test, ANOVA by hand, chi-square on counts, Pearson vs Spearman, one-sided tests.40 min
  46. Project: an exploratory analysis reportFive business questions answered with cleaning, joins, charts and a test, then written up.45 min
  47. Bandits: explore or exploitLearning from rewards with no labels: a running estimate for each option, why the greedy choice gets stuck, epsilon-greedy and an optimism bonus.35 min
  48. Decisions in sequence: states, rewards and valueA Markov decision process on a grid world: discounted return, the value of a state, the Bellman equation, and value iteration to the best policy.40 min
  49. Q-learning: learning values by doingA value for every action, learned from experience with no rules given: the update, a grid world you can watch, and why optimism explores better than chance.40 min
  50. Policy gradients: learning what to do directlyA policy as probabilities you can nudge: turn up what worked, measure it against a baseline, and see how the same update trains a language model.40 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write