Policy gradients: learning what to do directly
A policy as probabilities you can nudge: turn up what worked, measure it against a baseline, and see how the same update trains a language model.
What it covers6
Why it matters
Q-learning learns how good each action is and then picks the best. There is a more direct way: keep the policy itself as a set of adjustable numbers, try things, and turn up the probability of whatever worked. That is the method behind the reinforcement learning that trains language models on human preferences (RLHF) and on problems with checkable answers (Training reasoning models). A language model already is a policy: it gives a probability to every next word. This lesson builds the update on five headlines and a small grid, and the last part carries it across to models.
What you'll be able to do
- Represent a policy as probabilities from a softmax over preferences, and sample actions from it
- Apply the policy gradient update: raise the probability of an action in proportion to the return that followed it
- Explain why rewards are measured against a baseline, and show what goes wrong without one
- Say how the same update trains a language model: the model is the policy and each word is an action
Show the answer
Lock on to whichever headline it happens to show first, good or bad
If every outcome is rewarded, every choice is reinforced. Whatever the learner shows first gets more likely, so it is shown again, and reinforced again. The difference between 10 and 11 is real but tiny beside the 10 that everything earns. The fix, measuring each reward against what is normal, is the third part of this lesson.
The idea: preferences, probabilities and a nudge
Purpose. Keep a policy as numbers you can adjust, and raise the probability of whatever worked.
A Q-learning agent's policy is implied by its table: take the largest value. A policy gradient method stores the policy itself. For each action it keeps a number called a preference, and the softmax function from the softmax lesson turns the preferences into probabilities. The agent then samples its action from those probabilities.
Setup (runs before every cell) · 29 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import numpy as np true_rates = np.array([0.20, 0.50, 0.35, 0.65, 0.45]) def softmax_rows(x): e = np.exp(x - x.max(axis=1, keepdims=True)) return e / e.sum(axis=1, keepdims=True) def reinforce(baseline, bonus=0.0, alpha=0.3, steps=2000, players=500, seed=0): """500 players with five preferences each: clicks per step, and the probability each gives the best headline.""" rng = np.random.default_rng(seed) arms, rows = len(true_rates), np.arange(players) prefs, average = np.zeros((players, arms)), np.zeros(players) clicks, best = np.zeros(steps), np.zeros(steps) for t in range(steps): probs = softmax_rows(prefs) choice = (rng.random((players, 1)) > probs.cumsum(axis=1)).sum(axis=1) click = (rng.random(players) < true_rates[choice]).astype(float) reward = click + bonus average += (reward - average) / (t + 1) signal = reward - average if baseline else reward direction = -probs direction[rows, choice] += 1.0 prefs += alpha * signal[:, None] * direction clicks[t], best[t] = click.mean(), probs[:, true_rates.argmax()].mean() return clicks, best
import numpy as np def softmax(x): e = np.exp(x - x.max()) return e / e.sum() prefs = np.array([0.0, 1.0, 2.0]) # one preference per action probs = softmax(prefs) print("probabilities:", probs.round(3)) rng = np.random.default_rng(0) draws = rng.choice(3, size=1000, p=probs) # sample an action 1,000 times print("chosen in 1,000 draws:", np.bincount(draws, minlength=3))
The third action has the highest preference, so it gets a probability of 0.665 and is chosen 697 times in a thousand. The first is still chosen 84 times. A policy like this explores on its own: nothing is ever ruled out, and how often something is tried is set by its probability.
Learning will mean changing the preferences. That has three advantages worth knowing.
- It is smooth. A small change to a preference is a small change in behavior. A Q-table's policy can flip completely when one value overtakes another.
- It scales to huge action sets. A language model has tens of thousands of possible next words. It can't keep a separate table row for each, but it already outputs a probability for every one.
- The policy can be random on purpose. In rock-paper-scissors the best policy is to play each a third of the time. A probability can say that. "Take the largest Q-value" can't.
Show the answer
A third each, and adding 5 to all of them changes nothing
Softmax turns equal scores into equal probabilities: a third each. And it depends only on the differences between the scores, so adding the same number to every preference leaves every probability as it was.
Turn up what worked
Here is the update. The agent samples an action , and some return follows. It then changes every preference:
In plain words: (theta) is the preference for action and (pi) is its current probability. is 1 for the action that was taken and 0 for the others. So the action taken moves by , up if the return was positive. Every other action moves by , the opposite way. is the step size.
A good outcome makes the chosen action more likely and the rest less. A bad one, a negative , does the reverse. And the size of the push scales with the size of the return.
Uses the setup above, 29 lines.
import numpy as np def softmax(x): e = np.exp(x - x.max()) return e / e.sum() alpha = 0.5 prefs = np.array([0.0, 0.0, 0.0]) probs = softmax(prefs) action = 1 # the action that was sampled direction = -probs # 1 for the action taken, minus every probability direction[action] += 1.0 print("probabilities before:", probs.round(3)) print("direction: ", direction.round(3)) print("after a return of +1:", softmax(prefs + alpha * 1.0 * direction).round(3)) print("after a return of -1:", softmax(prefs + alpha * -1.0 * direction).round(3))
From three equal probabilities, a return of +1 after the middle action takes it from 0.333 to 0.452. A return of −1 takes it down to 0.233 and shares the difference between the other two.
Where does that direction come from? It is a gradient, in the sense of the gradient descent lesson: the slope of the log of the chosen action's probability. Multiplying it by the return and stepping along it, over many samples, climbs toward policies with a higher expected return. That is why these are called policy gradient methods. This simplest one has a name of its own, REINFORCE.
Now the five headlines from the bandit lesson, with a softmax policy and this update:
Uses the setup above, 29 lines.
import numpy as np true_rates = np.array([0.20, 0.50, 0.35, 0.65, 0.45]) def softmax_rows(x): e = np.exp(x - x.max(axis=1, keepdims=True)) return e / e.sum(axis=1, keepdims=True) def reinforce(baseline, bonus=0.0, alpha=0.3, steps=2000, players=500, seed=0): """500 players, each with five preferences. Returns clicks per step and the probability each gives the best headline.""" rng = np.random.default_rng(seed) arms, rows = len(true_rates), np.arange(players) prefs, average = np.zeros((players, arms)), np.zeros(players) clicks, best = np.zeros(steps), np.zeros(steps) for t in range(steps): probs = softmax_rows(prefs) # sample one headline per player choice = (rng.random((players, 1)) > probs.cumsum(axis=1)).sum(axis=1) click = (rng.random(players) < true_rates[choice]).astype(float) reward = click + bonus # each player's mean reward so far average += (reward - average) / (t + 1) signal = reward - average if baseline else reward direction = -probs # 1 for the arm shown, minus the probabilities direction[rows, choice] += 1.0 prefs += alpha * signal[:, None] * direction clicks[t], best[t] = click.mean(), probs[:, true_rates.argmax()].mean() return clicks, best clicks, best = reinforce(baseline=False) print("probability of the best headline at steps 1, 100, 500, 2000:", [round(float(best[t - 1]), 2) for t in (1, 100, 500, 2000)]) print("clicks per 1000, last 1000 visitors:", round(float(clicks[-1000:].sum())))
The players start with a fifth of their probability on the best headline. After 2,000 visitors they give it 0.95, and collect 639 clicks per thousand, close to the 650 that is the most possible. No table of values was kept, and no rule about exploring was written: the probabilities did both jobs.
(Ignore the baseline and bonus arguments for a moment. They are for the next part.)
import numpy as np probs = np.array([0.5, 0.3, 0.2]) action, G, alpha = 0, 2.0, 0.1 direction = -probs direction[action] += 1.0 print((alpha * G * direction).round(2))
Show the answer
[ 0.1 -0.06 -0.04]
The direction is 1 − 0.5 = 0.5 for the action taken and −0.3 and −0.2 for the others. Multiplied by α × G = 0.2, the changes are +0.1, −0.06 and −0.04. They add up to zero: probability moved toward the action that worked, away from the rest.
Show the answer
The same slope with its sign turned: y − π
Descending the cross-entropy loss −log π_a moves the scores along −(π − y) = y − π, which is [b = a] − π. So the policy gradient direction is exactly the slope of the log probability of the action taken. Climbing it, scaled by the return, raises the probability of actions that worked.
One policy gradient update
Write policy_update(prefs, probs, action, signal, alpha). prefs and probs are lists of the same length, action is the index of the action that was taken, and signal is the return (or advantage) that followed. Return a new list of preferences: the action taken moves by alpha * signal * (1 - probs[action]), and every other action b by -alpha * signal * probs[b]. Don't change the lists you were given.
Uses the setup above, 29 lines.
def policy_update(prefs, probs, action, signal, alpha): return list(prefs) print(policy_update([0.0, 0.0, 0.0], [1 / 3, 1 / 3, 1 / 3], 1, 1.0, 0.5))
The picture. One update, on three equal preferences. The middle action was sampled. After a return of +1 its probability rises from 0.333 to 0.452 and the other two share the loss. After a return of −1 the same push goes the other way. The numbers are the ones the cell above prints.
The three probabilities always add to 1: what one action gains, the others lose.
Write a sentence or two, then compare it with a good answer in the lesson.
In code: a baseline, many steps and PyTorch
Purpose. Fix what goes wrong when every reward is positive, then run the same update over a whole episode and in PyTorch.
Better than what?
The update pushes toward an action whenever the return is positive. That hides a trap, the one from the opening question. Take the same five headlines and add 10 to every reward: 10 for no click, 11 for a click. Nothing about the problem has changed. The best headline is still the best, by exactly as much.
Uses the setup above, 29 lines.
import numpy as np true_rates = np.array([0.20, 0.50, 0.35, 0.65, 0.45]) def softmax_rows(x): e = np.exp(x - x.max(axis=1, keepdims=True)) return e / e.sum(axis=1, keepdims=True) def reinforce(baseline, bonus=0.0, alpha=0.3, steps=2000, players=500, seed=0): rng = np.random.default_rng(seed) arms, rows = len(true_rates), np.arange(players) prefs, average = np.zeros((players, arms)), np.zeros(players) clicks, best = np.zeros(steps), np.zeros(steps) for t in range(steps): probs = softmax_rows(prefs) choice = (rng.random((players, 1)) > probs.cumsum(axis=1)).sum(axis=1) click = (rng.random(players) < true_rates[choice]).astype(float) reward = click + bonus # each player's mean reward so far average += (reward - average) / (t + 1) signal = reward - average if baseline else reward direction = -probs direction[rows, choice] += 1.0 prefs += alpha * signal[:, None] * direction clicks[t], best[t] = click.mean(), probs[:, true_rates.argmax()].mean() return clicks, best for name, baseline, bonus in (("no baseline, rewards 0 or 1 ", False, 0.0), ("no baseline, rewards 10 or 11", False, 10.0), ("baseline, rewards 0 or 1 ", True, 0.0), ("baseline, rewards 10 or 11", True, 10.0)): clicks, best = reinforce(baseline, bonus) print(name, "| clicks per 1000, last 1000 visitors:", round(float(clicks[-1000:].sum())), "| probability of the best headline at the end:", round(float(best[-1]), 2))
Look at the second row. With rewards of 10 and 11 the learner collapses: 444 clicks, about what choosing at random gives, and only 0.23 of its probability on the best headline. Every choice earned at least 10, so every choice was reinforced hard, and each player locked on to whatever it tried first.
The last two rows fix it with one subtraction. In place of the reward, use the reward minus a baseline: a measure of what is normal. Here the baseline is the player's own average reward so far, the latest one included. The signal is now "better or worse than usual": positive for a click, negative for none, whatever the rewards' scale. With it, the learner gets 648 clicks in both versions of the problem, to the last digit.
The four runs as a picture. The probability each player gives the best headline, over the 2,000 visitors. The setup holds the reinforce function from the cells above:
Uses the setup above, 29 lines.
import matplotlib.pyplot as plt runs = [("no baseline, rewards 0 or 1", False, 0.0), ("no baseline, rewards 10 or 11", False, 10.0), ("baseline, rewards 0 or 1", True, 0.0), ("baseline, rewards 10 or 11", True, 10.0)] for label, baseline, bonus in runs: clicks, best = reinforce(baseline, bonus) plt.plot(best, label=label) plt.xlabel("visitor") plt.ylabel("probability of the best headline") plt.legend() plt.show()
The two baseline curves lie on top of each other: shifting every reward by 10 changes nothing once the baseline is subtracted. The no-baseline curve with rewards 10 or 11 stays near the bottom.
Show the answer
No baseline: every action now earns a large positive reward and is reinforced
Shifting every reward by 100 doesn't change which action is best. But an update that multiplies by the raw reward now pushes hard toward whatever was sampled, and the difference of 1 is lost. Subtracting a baseline, such as the average reward, restores the signal.
The difference between what happened and what was expected has a name: the advantage.
In plain words: is the advantage, the return that followed the action, and the baseline, the return you'd normally expect. An advantage above zero means the action did better than usual and should become more likely. Below zero, less likely.
Even when rewards are well scaled, a baseline helps. Compare the first and third rows: 639 clicks without, 648 with. Subtracting the expected part removes noise from every update without changing where the updates lead on average, so the learner steps more surely.
This is the same error you have seen in every lesson of this track: what happened, minus what was expected.
G, baseline = 10.0, 10.4 advantage = G - baseline print(round(advantage, 1), "more likely" if advantage > 0 else "less likely")
Show the answer
-0.4 less likely
The return of 10 is below the usual 10.4, so the advantage is −0.4 and the update makes the action less likely. Without the baseline the raw return, +10, would have made it much more likely, which is how the learner in the second row went wrong.
Advantages from a list of returns
Write advantages(returns). Go through the list in order. For each return, the baseline is the mean of the returns so far, this one included, as in the lesson's code, and the advantage is the return minus that baseline. So the first advantage is always 0: with nothing to compare it to, the first return counts as exactly usual. Return the list of advantages, each rounded to 3 places.
Uses the setup above, 29 lines.
def advantages(returns): return list(returns) print(advantages([10, 11, 10, 11, 11]))
Many steps, and the road to language models
So far each action had its own immediate reward. In a task with many steps, one change is needed: the signal for an action is the return from that step onward, since an action can't be credited for rewards that came before it. Here is the grid world again, learned by policy gradient with a baseline for each square.
Uses the setup above, 29 lines.
import numpy as np ROWS, COLS, WALL, START = 3, 4, (1, 1), (2, 0) ENDS = {(0, 3): 1.0, (1, 3): -1.0} MOVES = {"up": (-1, 0), "down": (1, 0), "left": (0, -1), "right": (0, 1)} ACTIONS = list(MOVES) STATES = [(r, c) for r in range(ROWS) for c in range(COLS) if (r, c) != WALL] def step(state, action): r, c = state[0] + MOVES[action][0], state[1] + MOVES[action][1] nxt = (r, c) if 0 <= r < ROWS and 0 <= c < COLS and (r, c) != WALL else state return nxt, ENDS.get(nxt, -0.04) def softmax(x): e = np.exp(x - x.max()) return e / e.sum() def reinforce(episodes, alpha=0.2, gamma=0.9, seed=0): rng = np.random.default_rng(seed) prefs = {s: np.zeros(4) for s in STATES} # four preferences in every square # the average return seen from each square: the baseline value = {s: 0.0 for s in STATES} seen = {s: 0 for s in STATES} returns = [] for episode in range(episodes): s, path = START, [] # play one episode by sampling from the policy while s not in ENDS and len(path) < 60: a = rng.choice(4, p=softmax(prefs[s])) nxt, reward = step(s, ACTIONS[a]) path.append((s, a, reward)) s = nxt G = 0.0 # work backward: G is the return from each step onward for s, a, reward in reversed(path): G = reward + gamma * G advantage = G - value[s] seen[s] += 1 value[s] += (G - value[s]) / seen[s] direction = -softmax(prefs[s]) direction[a] += 1.0 prefs[s] += alpha * advantage * direction returns.append(sum(reward for s, a, reward in path)) return prefs, returns prefs, returns = reinforce(1000) print("mean return by hundred episodes:", [round(float(np.mean(returns[i:i + 100])), 2) for i in range(0, 1000, 100)]) print("policy at the start square:", dict(zip(ACTIONS, softmax(prefs[START]).round(2).tolist())))
The first hundred episodes average 0.30 and the second hundred 0.79. By the end the agent averages 0.83, beside a best possible 0.84, and at the start square it goes up with probability 0.97. It found the route Q-learning found, by a different road: no table of action values, only probabilities, pushed toward whatever turned out better than the square's average.
rewards, gamma = [0, 0, 1], 0.9 G, out = 0.0, [] for reward in reversed(rewards): G = reward + gamma * G out.append(round(G, 2)) print(out[::-1])
Show the answer
[0.81, 0.9, 1.0]
Working backward: the last move's return is 1.0. The move before it gets 0 + 0.9 × 1.0 = 0.9, and the first gets 0 + 0.9 × 0.9 = 0.81. Each move is credited with what followed it, discounted by how long it took to arrive.
The same update in PyTorch. In a library, nobody writes the direction by hand. The policy gives a log probability for the action taken, and the loss is that log probability times the advantage, with a minus sign so that a descent step climbs the probability. Backpropagation then produces the direction. The gradient on the preferences is the negative of the direction from the cell above:
Uses the setup above, 29 lines.
import torch from torch.distributions import Categorical prefs = torch.zeros(3, requires_grad=True) dist = Categorical(logits=prefs) action, advantage = torch.tensor(1), 1.0 loss = -dist.log_prob(action) * advantage loss.backward() print([round(g, 3) for g in (-prefs.grad).tolist()])
The printed list is the direction: +0.667 for the action taken and −0.333 for each of the others, exactly the numbers behind the first picture, with α = 1. The browser runs a teaching subset of PyTorch, the same code runs on a real install, and seeded random numbers differ between them, so only the structure is compared here.
Worked example: one episode's update, by hand
Follow a single short episode on a corridor of two squares, each with two actions, and compute every number of the update: the returns from each step, the advantages against the squares' baselines, and the new probabilities.
Uses the setup above, 29 lines.
import numpy as np def softmax(x): e = np.exp(x - x.max()) return e / e.sum() alpha, gamma = 0.5, 0.9 # actions: 0 = left, 1 = right prefs = {"first": np.array([0.0, 0.0]), "second": np.array([0.0, 0.0])} # the average return seen from each square so far baseline = {"first": 0.2, "second": 0.5} # went right (reward 0), then right again (reward 1, the end) path = [("first", 1, 0.0), ("second", 1, 1.0)] G = 0.0 for square, action, reward in reversed(path): G = reward + gamma * G advantage = G - baseline[square] direction = -softmax(prefs[square]) direction[action] += 1.0 prefs[square] = prefs[square] + alpha * advantage * direction print(square, "| return from here", round(G, 2), "| advantage", round(advantage, 2), "| probability of right is now", round(float(softmax(prefs[square])[1]), 3))
The loop runs backward, so the second square comes first. Its return is the final reward, 1.0, half a point above that square's baseline of 0.5, and the probability of going right there rises from 0.5 to 0.562. Then the first square: its return is 0 + 0.9 × 1.0 = 0.9. That move's own reward was 0, but it is credited with what it led to, and 0.9 is well above its baseline of 0.2, so its probability of going right rises more, to 0.587. One episode moved both squares a little. Thousands of episodes, each a small push toward what did better than expected, are the whole of the method.
Write a sentence or two, then compare it with a good answer in the lesson.
In practice: the same update on a language model
Purpose. Map the lesson onto a language model, and see the one addition that real training methods make.
The same update, on a language model
Everything is now in place to see how a language model is trained with rewards, the subject of several lessons in the phase on modern LLM engineering.
| In this lesson | In a language model |
|---|---|
| The state | The text so far |
| An action | The next word (token) |
| The policy's preferences | The model's output scores, set by its weights |
| An episode | One whole answer |
| The reward | A score for the answer: from people's ratings, a model trained to predict them, or a checker |
| The update | Shift the weights so that the words of better-than-expected answers become more likely |
The model writes an answer by sampling one word after another: an episode. The answer gets a score. The gradient that makes the sampled words more probable is computed by backpropagation (Backpropagation), in place of this lesson's one-line direction. It is multiplied by the advantage, and the weights take a small step. Methods such as PPO (RLHF) are this update with one main addition: a limit on how far the policy may move in a single step, because a policy that changes too fast collects its next batch of experience from somewhere it has never been.
The one addition real methods make
Write a sentence or two, then compare it with a good answer in the lesson.
Practice what you just read
- 8 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 11 review cards for spaced repetition
Try a quiz question
What does a policy gradient method adjust as it learns?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 3 · about 39 hours of study
Machine learning
Learning from data, and knowing when it worked
50 lessons 1 free
- What machine learning is FreePrograms that learn rules from examples instead of being told them.30 min
- Features and labelsSupervised learning: inputs, targets and predictions.30 min
- Train, validation and test setsHold data back so you can trust your results.38 min
- Preparing dataClean, encode and scale data before a model sees it.40 min
- Linear regressionFit a line, and predict numbers from features.42 min
- Loss functionsOne number that says how wrong the model is.35 min
- Training with gradient descentAdjust weights step by step to reduce the loss.42 min
- Learning rateStep size, convergence and divergence.35 min
- Logistic regressionPredict probabilities for yes/no outcomes.42 min
- Multiclass classificationSoftmax and cross-entropy for more than two classes.42 min
- The same three models in PyTorchA line, logistic regression and softmax regression fitted both ways: the loss, the gradient and the update opened up, with weight decay and mini-batches.35 min
- Precision, recall and F1Accuracy lies; these don't.42 min
- ROC curves and AUCJudge a classifier across every threshold.30 min
- Calibration: can the probability be trusted?Bands of predicted risk against what happened, the Brier score, why naive Bayes is over-confident, and fixing it on rows the model did not fit.30 min
- Overfitting and underfittingMemorizing versus learning.30 min
- RegularizationPenalize complexity so models generalize.40 min
- Bias and varianceThe two ways a model can be wrong.30 min
- Cross-validationReuse limited data for more reliable estimates.42 min
- Imbalanced classes in depthRare classes measured: why rebalancing mostly moves the threshold, correcting inflated probabilities, average precision against AUC, why the count of rare cases matters, and thresholds from costs.45 min
- Naive Bayes: classifying with countsA classifier trained by counting: Bayes' rule over words, the naive assumption, log scores, smoothing for unseen words, why it is overconfident, and when it beats logistic regression.45 min
- Nearest neighborsPredict from the most similar examples.35 min
- Decision treesLearn a flowchart of yes/no questions.42 min
- Random forests and boostingMany weak models voting beat one strong one.42 min
- Gradient boosting in depthBoosting built from scratch: trees fit to the gradient of the loss, depth and interactions, learning rate against number of trees, early stopping, and what XGBoost and LightGBM add.45 min
- Support vector machinesThe widest margin, support vectors, the hinge loss and C, kernels for curved boundaries, and when an SVM fits.40 min
- Recommender systemsShrunk averages, item-to-item collaborative filtering, matrix factorization, and cold start.40 min
- Project: a price modelMultiple regression on log prices, one-hot categories, percentage effects and an extrapolation flag.45 min
- Feature engineeringTurn raw data into signals a model can use.35 min
- The scikit-learn workflowFit, predict and evaluate with the standard ML library.45 min
- Explaining a model: what it uses, and why this predictionPermutation importance on held-back rows, a linear model's contributions for one customer, partial dependence, and what none of them can say.35 min
- Capstone: churn predictionPredict which customers will leave, and measure it properly.45 min
- ClusteringGroup similar items without labels (k-means).35 min
- Clusters of any shape: DBSCAN and hierarchiesWhy k-means needs round clusters, DBSCAN's core points and noise, and hierarchical clustering with its linkages.40 min
- Anomaly detectionMasking, robust scores with the median and MAD, baselines from the past, isolation forests and thresholds from a review budget.40 min
- Dimensionality reductionVisualize and compress high-dimensional data.35 min
- Cleaning messy dataDuplicates, junk rows, text numbers, mixed dates and missing values that mean something.45 min
- Grouping, pivoting and reshapingNamed aggregation, pivot tables, transform, running totals and groups too small to rank.45 min
- Data in pieces: partitions and mergeable summariesSplit, summarize and combine; which statistics merge exactly, merging variances, sketches for the rest, and partitioning well.30 min
- Combining tables with mergeJoin on keys safely, stack tables, and divide by the right denominator.44 min
- Time series with pandasResample, smooth, compare across years, and test whether one series leads another.45 min
- Forecasting: baselines, lag features and backtestsThree baselines, lag features that never see their own future, a walk-forward backtest, and a range around the number.40 min
- Text mining: TF-IDF, search and labelsBag of words and TF-IDF, similarity search and its blind spots, text classifiers and topics, and when LLM labels are worth it.35 min
- Location dataHaversine distances, nearest places, GeoJSON, point in polygon and simple maps.40 min
- Comparing groups: t-testsWelch's t-test, pooled vs average rates, and a control group for before-and-after changes.40 min
- More tests: groups, categories, relationshipsChoose a test, ANOVA by hand, chi-square on counts, Pearson vs Spearman, one-sided tests.40 min
- Project: an exploratory analysis reportFive business questions answered with cleaning, joins, charts and a test, then written up.45 min
- Bandits: explore or exploitLearning from rewards with no labels: a running estimate for each option, why the greedy choice gets stuck, epsilon-greedy and an optimism bonus.35 min
- Decisions in sequence: states, rewards and valueA Markov decision process on a grid world: discounted return, the value of a state, the Bellman equation, and value iteration to the best policy.40 min
- Q-learning: learning values by doingA value for every action, learned from experience with no rules given: the update, a grid world you can watch, and why optimism explores better than chance.40 min
- Policy gradients: learning what to do directlyA policy as probabilities you can nudge: turn up what worked, measure it against a baseline, and see how the same update trains a language model.40 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write