Index Ai Academy
Phase 3 · Reinforcement learning

Q-learning: learning values by doing

A value for every action, learned from experience with no rules given: the update, a grid world you can watch, and why optimism explores better than chance.

About 40 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

Value iteration found the best route through the grid world, and it cheated: it was handed the rules and never took a step. An agent in the real world has no rule book. A program learning a game, a robot learning to walk, a model learning which answers people prefer: each can only try something and see what comes back. Q-learning is the classic method for that. It joins Bandits and Decisions in sequence: the bandit's running estimate, updated from one reward at a time, and the Bellman equation's look ahead to the next state. The result learns the best policy by acting, with no model of the world at all.

What you'll be able to do

  • Say what a Q-value is, and why an agent that doesn't know the rules needs one per action
  • Apply the Q-learning update to one move: old estimate, plus a step toward reward plus discounted best next value
  • Train an agent on a grid world from experience alone and read the policy it learned
  • Show a case where random exploring fails, and explain why starting estimates high fixes it
Quick check
A guess before you start. An agent knows the value of every square in a grid world, but not what its moves do: it has no idea which square "up" leads to. Can it choose a good move?
  1. No: values of squares don't help unless it knows which square each move reaches
  2. No: an agent can never act well in a world it hasn't been told about
  3. Yes: it should move to whichever neighboring square has the highest value
  4. Yes: knowing the value of the square it stands on is enough to choose
Show the answer

No: values of squares don't help unless it knows which square each move reaches

"Move to the best neighbor" needs to know which move goes to which neighbor, and that is exactly the rule book it lacks. The fix is to learn a value for each action directly: not "how good is this square" but "how good is it to go up from this square". Then choosing is just comparing numbers it has.

The idea: a value for every action

Purpose. Learn how good each move is from experience alone.

The V(s)V(s) of Decisions in sequence valued states, and needed the rules to turn those values into moves. Without the rules, the agent needs a different table.

Q(s,a)Q(s, a) is the return the agent can expect if it takes action aa in state ss and acts as well as it can from then on: a Q-value, one number for every state and action together. To choose a move, look along the row for the current state and take the action with the largest number.

How does the table get filled? By the bandit's update, with one change. The bandit moved its estimate toward the reward it had just received. Here the reward is only the start of what the move earns, so the agent moves its estimate toward the reward plus what it now believes the next state is worth:

Q(s,a)←Q(s,a)+α(r+γmax⁡a′Q(s′,a′)−Q(s,a))Q(s, a) \leftarrow Q(s, a) + \alpha \big( r + \gamma \max_{a'} Q(s', a') - Q(s, a) \big)

In plain words: the agent was in state ss, took action aa, received reward rr and arrived in state s′s'. The target is rr plus gamma times the largest Q-value in the new state's row, max⁡a′Q(s′,a′)\max_{a'} Q(s', a'): the Bellman equation, with the agent's own current estimates standing in for the truth. The bracket is the error, target minus old estimate. α\alpha (alpha) is the learning rate, the fraction of that error to move by.

One update, by hand:

alpha, gamma = 0.5, 0.9
Q = {
    "A": {"left": 0.0, "right": 0.2},
    "B": {"left": 0.1, "right": 0.6},
}

# The agent is in A, goes right, gets a reward of -0.04 and finds itself in B.
s, a, r, s_next = "A", "right", -0.04, "B"
target = r + gamma * max(Q[s_next].values())
print("target:", round(target, 3))
print("error: ", round(target - Q[s][a], 3))
Q[s][a] += alpha * (target - Q[s][a])
print("new Q(A, right):", round(Q[s][a], 3))
It runs right here in your browser, with nothing to install.

The best the agent believes it can do from B is 0.6, so going right from A looks worth −0.04 + 0.9 × 0.6 = 0.5. Its old estimate was 0.2. It moves halfway: 0.35. Nothing else in the table changed.

Quick check
An agent in state X takes "up", receives +1 and the episode ends. Its estimate Q(X, up) was 0.4 and alpha is 0.5. What does this print?
alpha, gamma = 0.5, 0.9
q_old, reward = 0.4, 1.0
target = reward                    # the episode ended: nothing follows
print(round(q_old + alpha * (target - q_old), 2))
  1. 0.7
  2. 1.0
  3. 1.3
  4. 0.85
Show the answer

0.7

The episode ended, so the target is the reward alone, 1.0. The error is 1.0 − 0.4 = 0.6, and half of it is 0.3: the estimate moves from 0.4 to 0.7.

Drill

One Q-learning update

Write q_update(Q, s, a, reward, s_next, alpha, gamma). Q maps each state to a dictionary of action values. Change Q[s][a] in place: move it by alpha times the error, where the target is reward + gamma * (the largest value in Q[s_next]). If s_next is None, the episode ended and the target is reward alone. Return the error (target minus the old estimate).

def q_update(Q, s, a, reward, s_next, alpha, gamma):
    return 0.0


Q = {"A": {"left": 0.0, "right": 0.2}, "B": {"left": 0.1, "right": 0.6}}
print(round(q_update(Q, "A", "right", -0.04, "B", 0.5, 0.9), 3), Q["A"])
In the lesson, your code is checked as you write it.

The picture. The update from the cell above, drawn as bars. The estimate Q(A, right) starts at 0.2. The agent's experience gives a target, −0.04 + 0.9 × 0.6 = 0.5. The estimate then moves a fraction alpha = 0.5 of the way toward it.

Try it now

The same cell prints target 0.5, error 0.3 and new Q 0.35.

In your own words
In your own words: how is the Q-learning update like the bandit's update, and what does it add?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: the whole agent, and what it learns

Purpose. Train an agent that knows no rules, compare it with value iteration, and find a case where random exploring fails.

Code flow. The update, line by line, on the numbers from the picture. Step through it and watch the error and the new estimate.

Watching it learn

Here is the whole agent, on the grid world from Decisions in sequence. It chooses moves epsilon-greedily, as the bandit player did, and applies the update after every one. The only thing it ever asks of the world is step: "I did this; what happened?"

import numpy as np

ROWS, COLS, WALL, START = 3, 4, (1, 1), (2, 0)
ENDS = {(0, 3): 1.0, (1, 3): -1.0}
MOVES = {"up": (-1, 0), "down": (1, 0), "left": (0, -1), "right": (0, 1)}
ACTIONS = list(MOVES)
STATES = [(r, c) for r in range(ROWS) for c in range(COLS) if (r, c) != WALL]
ARROW = {"up": "^", "down": "v", "left": "<", "right": ">"}

def step(state, action):
    r, c = state[0] + MOVES[action][0], state[1] + MOVES[action][1]
    nxt = (r, c) if 0 <= r < ROWS and 0 <= c < COLS and (r, c) != WALL else state
    return nxt, ENDS.get(nxt, -0.04)

def q_learning(episodes, alpha=0.5, gamma=0.9, epsilon=0.1, seed=0):
    rng = np.random.default_rng(seed)
    Q = {s: {a: 0.0 for a in ACTIONS} for s in STATES}
    returns = []
    for episode in range(episodes):
        s, total, moves = START, 0.0, 0
        while s not in ENDS and moves < 100:
            if rng.random() < epsilon:
                a = ACTIONS[rng.integers(4)]                          # explore
            else:
                a = max(ACTIONS, key=lambda x: Q[s][x])               # exploit
            nxt, reward = step(s, a)
            target = reward if nxt in ENDS else reward + gamma * max(Q[nxt].values())
            Q[s][a] += alpha * (target - Q[s][a])
            s, total, moves = nxt, total + reward, moves + 1
        returns.append(total)
    return Q, returns

Q, returns = q_learning(300)
for a, b in ((0, 10), (10, 20), (20, 50), (200, 300)):
    print("episodes", a + 1, "to", b, "| mean return",
          round(float(np.mean(returns[a:b])), 3))
print("Q at the start square:", {a: round(v, 3) for a, v in Q[START].items()})
for r in range(ROWS):
    arrows = " ".join(
        "#" if (r, c) == WALL
        else "G" if ENDS.get((r, c)) == 1.0
        else "P" if (r, c) in ENDS
        else ARROW[max(ACTIONS, key=lambda a: Q[(r, c)][a])]
        for c in range(COLS))
    values = "  ".join(
        "  ##  " if (r, c) == WALL
        else " GOAL " if ENDS.get((r, c)) == 1.0
        else " PIT  " if (r, c) in ENDS
        else f"{max(Q[(r, c)].values()):6.3f}"
        for c in range(COLS))
    print(arrows, "  |  ", values)

The first ten episodes average −0.132: the agent wanders, bumps into walls and sometimes falls in the pit. By episodes 11 to 20 it averages 0.824, close to the 0.84 of the shortest route. It learned the task in about a dozen tries.

Quick check
The agent's first ten episodes average −0.132 and its next ten average 0.824. Its code is the same in both. What changed?
  1. Its epsilon: it stopped exploring once it had found the goal
  2. Its learning rate: alpha grows as the agent gains experience
  3. The world: the pit's penalty shrinks as the episodes go by
  4. Its Q-table: the estimates now point its greedy moves toward the goal
Show the answer

Its Q-table: the estimates now point its greedy moves toward the goal

Everything the agent has learned lives in the table. At first every entry is 0 and its greedy choice is no better than chance. After a dozen episodes the entries along the route are large and the entries beside the pit are negative, so the same line of code, "take the action with the largest Q-value", now walks to the goal.

Now compare the grid at the bottom of that output with value iteration's answer from Decisions in sequence. The start square reads 0.519 in both. The squares along the route read 0.621, 0.734 and 0.860 in both. The agent has recovered the exact values, having never seen the rules.

Two squares differ. The middle of the left column reads 0.614 where the true value is 0.621, and the bottom-right corner 0.578 where it is 0.621. The agent rarely goes there, so those estimates have had few updates. Q-learning is only accurate where it has been. That is a weakness and also its strength: it spends its effort on the states that matter for the task, and can work in worlds with far too many states to sweep.

Quick check
The bottom-right corner reads 0.578 where the true value is 0.621. What is the most likely reason?
  1. The agent's code has a bug that only shows in the corners
  2. Alpha is larger in the corners than on the route
  3. The agent rarely goes there, so its estimate has had few updates
  4. The corner's true value changes between episodes as it learns
Show the answer

The agent rarely goes there, so its estimate has had few updates

An estimate improves only when the agent takes that action in that state. The route to the goal is traveled in almost every episode, so its values settle. The corner is reached only by an occasional exploring move.

Notice also that the average return near the end, 0.724, is below the 0.84 of a perfect run. The policy it learned is right. But it still explores one move in ten, and now and then a random move drops it in the pit.

Go deeper Learning one policy while following another

The target uses the best next action's value, whatever the agent does next. So Q-learning learns about the greedy policy while behaving epsilon-greedily: off-policy learning. Its cousin SARSA uses the action really taken next, and so learns the value of its own exploring behavior: on-policy. Beside a pit, SARSA learns to keep its distance.

Quick check
After training, an agent's Q-table has this row for one state. Which move does the greedy policy take, and what is the state's value? What does this print?
row = {"up": 0.49, "down": 0.423, "left": 0.426, "right": 0.519}
print(max(row, key=row.get), max(row.values()))
  1. right 0.49
  2. up 0.49
  3. down 0.423
  4. right 0.519
Show the answer

right 0.519

The greedy policy takes the action with the largest Q-value, here right at 0.519. And the value of a state under that policy is the largest Q-value in its row: V(s) is the maximum of Q(s, a) over the actions.

Quick check
From the lesson on decisions in sequence: value iteration needs something that Q-learning does without. What is it?
  1. A table with one entry for each state of the world
  2. The rules: where every action leads from every state, and what it pays
  3. Rewards that tell the agent how well it is doing
  4. A discount factor to make later rewards count for less
Show the answer

The rules: where every action leads from every state, and what it pays

Value iteration computes with the rules, calling them for every state and action without moving. Q-learning never sees them. It learns from the moves it actually makes: state, action, reward, next state.

Drill

Test what was learned, with exploring off

While learning, the agent's returns are dragged down by its own exploring. To see what it has learned, follow its best moves with no exploring and no updates. Write follow(Q, world, start, max_moves). world(state, action) returns (reward, next_state), with next_state equal to None when the episode ends. From start, keep taking the action with the largest value in Q[state] (the first listed on a tie) until the episode ends or max_moves moves have been made. Return a tuple: the total reward collected, and the number of moves made. Don't change Q.

def follow(Q, world, start, max_moves):
    return 0.0, 0


def corridor(state, action):
    if action == "left":
        return (0.2, None) if state == 0 else (-0.04, state - 1)
    return (1.0, None) if state == 2 else (-0.04, state + 1)


Q = {0: {"left": 0.2, "right": 0.7}, 1: {"left": 0.1, "right": 0.86},
     2: {"left": 0.5, "right": 1.0}}
print(follow(Q, corridor, 0, 20))
In the lesson, your code is checked as you write it.

When random exploring isn't enough

Add one thing to the world: a small prize worth 0.2, in the square right next to the start. Taking it ends the episode. The goal, five moves away, is still worth far more: 0.519 from the start, as you have seen. Will the agent find it?

Try it. The agent below starts with the settings from the cell above.

The widget shows one agent. Here are twenty of each kind, with different seeds, 500 episodes each:

import matplotlib.pyplot as plt
import numpy as np

ROWS, COLS, WALL, START = 3, 4, (1, 1), (2, 0)
# the goal, the pit, and a small prize beside the start
ENDS = {(0, 3): 1.0, (1, 3): -1.0, (2, 1): 0.2}
MOVES = {"up": (-1, 0), "down": (1, 0), "left": (0, -1), "right": (0, 1)}
ACTIONS = list(MOVES)
STATES = [(r, c) for r in range(ROWS) for c in range(COLS) if (r, c) != WALL]

def step(state, action):
    r, c = state[0] + MOVES[action][0], state[1] + MOVES[action][1]
    nxt = (r, c) if 0 <= r < ROWS and 0 <= c < COLS and (r, c) != WALL else state
    return nxt, ENDS.get(nxt, -0.04)

def q_learning(episodes, epsilon, start_value, seed, alpha=0.5, gamma=0.9):
    rng = np.random.default_rng(seed)
    Q = {s: {a: start_value for a in ACTIONS} for s in STATES}
    returns = []
    for episode in range(episodes):
        s, total, moves = START, 0.0, 0
        while s not in ENDS and moves < 100:
            a = (ACTIONS[rng.integers(4)] if rng.random() < epsilon
                 else max(ACTIONS, key=lambda x: Q[s][x]))
            nxt, reward = step(s, a)
            target = reward if nxt in ENDS else reward + gamma * max(Q[nxt].values())
            Q[s][a] += alpha * (target - Q[s][a])
            s, total, moves = nxt, total + reward, moves + 1
        returns.append(total)
    return Q, returns

def where_it_ends(Q):
    s, moves = START, 0
    while s not in ENDS and moves < 30:
        s, moves = step(s, max(ACTIONS, key=lambda a: Q[s][a]))[0], moves + 1
    return s

for name, epsilon, start_value in (("epsilon 0.1, estimates start at 0", 0.1, 0.0),
                                   ("epsilon 0.5, estimates start at 0", 0.5, 0.0),
                                   ("epsilon 0,   estimates start at 1", 0.0, 1.0)):
    runs = [q_learning(500, epsilon, start_value, seed) for seed in range(20)]
    curve = np.mean([returns for Q, returns in runs], axis=0)
    plt.plot(curve.reshape(-1, 10).mean(axis=1), label=name)
    found = sum(where_it_ends(Q) == (0, 3) for Q, returns in runs)
    print(name, "| reaches the goal in", found,
          "of 20 runs | mean return, last 100 episodes:",
          round(float(np.mean([np.mean(returns[-100:]) for Q, returns in runs])), 3))
plt.xlabel("episodes, in tens")
plt.ylabel("mean return")
plt.legend()
plt.show()

With epsilon 0.1 and estimates starting at 0, none of the twenty agents learns the way to the goal. All of them take the prize, for a return of about 0.2. Turning exploration up to one move in two only gets 5 of 20 there, and those agents then throw away half their moves.

The third row has no random exploring at all, and all twenty find the goal and collect 0.84.

Here is why. Every estimate starts at 1, higher than any real return in this world. Whatever the agent tries turns out worse than promised, so that estimate drops, and some other, untried action is now the highest in the row. The agent is drawn to whatever it hasn't tried, square after square, until it has tried everything and its estimates are honest. This is the bandit lesson's optimism in the face of uncertainty again, and it explores far more systematically than a coin flip does.

The lesson to take: exploring isn't one dial to turn up. When the reward you want lies many steps beyond a smaller one, how an agent explores matters more than how often.

Quick check
Every Q-value starts at 1.0. The agent takes "right" from the start, receives 0.2, and the episode ends. Alpha is 0.5. What does this print: the new estimate for right, and the move it prefers next time?
row = {"up": 1.0, "down": 1.0, "left": 1.0, "right": 1.0}
row["right"] += 0.5 * (0.2 - row["right"])
print(round(row["right"], 2), max(row, key=row.get))
  1. 0.6 right
  2. 0.6 up
  3. 0.2 up
  4. 1.2 right
Show the answer

0.6 up

The target is 0.2, since the episode ended. The error is 0.2 − 1.0 = −0.8, and half of it takes right down to 0.6. The three untried moves still stand at 1.0, so next time the agent prefers one of those: up is the first.

Worked example

Worked example: the first two episodes, move by move

The curve above showed the agent going from hopeless to capable in a dozen episodes. Look at how: trace a tiny corridor, where the only reward is at the far end, and watch the reward's news travel back one square per episode.

alpha, gamma = 0.5, 0.9
# squares 0, 1, 2, 3 in a row; square 4 is the goal
Q = {square: {"right": 0.0} for square in range(4)}

def episode(n):
    # walk right from square 0 to the goal
    for s in range(4):
        reward = 1.0 if s == 3 else 0.0
        target = reward if s == 3 else reward + gamma * Q[s + 1]["right"]
        Q[s]["right"] += alpha * (target - Q[s]["right"])
    print("after episode", n, [round(Q[s]["right"], 3) for s in range(4)])

for n in range(1, 7):
    episode(n)
print("true values:   ", [round(gamma ** (3 - s), 3) for s in range(4)])
In the lesson you can change this and run it.

After the first episode only the square beside the goal has learned anything: it received the reward directly. In the second episode the square before it looks ahead, sees a neighbor now worth something, and picks up its first value. Each episode carries the news one square further back, which is why the last line's true values, 0.729, 0.81, 0.9 and 1.0, fill in from the right. This is the same spreading you watched in value iteration's sweeps, but driven by the agent's own walks. It is also why a reward far from the start, like the goal beyond the prize, is so slow to be noticed.

In your own words
Why did starting every estimate high make the agent explore, with no random moves at all?

Write a sentence or two, then compare it with a good answer in the lesson.

In practice: when a table is not enough

Purpose. See what replaces the table when states are too many to list, and what to watch in a real system.

What a table can't hold

Q-learning with a table has one hard limit: it needs a row for every state. This world has 11 states. Chess has about 10 to the power 44 positions, far more than any table could hold. A robot's camera image, or the text of a conversation so far, is a state you will never see twice.

The way out is the idea behind every model in this course: don't store an answer for each input, learn a function that produces one. Replace the table with a model that takes a description of the state and outputs a Q-value for each action, and train it so that its outputs move toward the same targets, r+γmax⁡Q(s′,a′)r + \gamma \max Q(s', a'). States that look alike then share what was learned. When the function is a neural network (Multi-layer networks), this is deep Q-learning, the method that learned to play Atari video games from the screen alone.

The table version is still the right place to have started. The update rule, the target, the trade-off between exploring and exploiting, and the trap of settling for a nearby reward all carry over unchanged.

The line that matters is the loss. The table moved its entry a fraction of the error. The network moves its weights down the gradient of the squared error, which is the same step in a form that generalizes. Deep Q-learning adds a replay buffer of old experience and a slowly-updated copy of the network for the targets, because a target that moves while it is learned is unstable.

Where this turns up in an engineer's week. Game-playing and control agents built on Q-networks; environments with a step function, like the one in the gymnasium library, where env.step(action) returns the next state, a reward and whether the episode ended, exactly the quantities this lesson used; evaluations of agents that act in a loop. The recurring problem is the one in the prize example: an agent that settles for a small reward it found early. When you see an agent plateau, look at how it explores before you change the learning rate.

On the job: run a trained agent with exploring switched off and no updates, as the follow drill did, and report that return. The return while learning understates what it has learned.

Goes deeper in

In your own words
A team trains a Q-learning agent on a task and it settles on a small, easy reward, ignoring a much larger one that takes many steps to reach. What would you try, and why?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 4 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 12 review cards for spaced repetition

Try a quiz question

What does Q(s, a) estimate?

  1. The return from taking action a in state s and acting well afterward
  2. The reward received immediately after taking action a in state s
  3. The chance that action a leads from state s to the goal
  4. The number of times the agent has taken action a in state s

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 3 · about 39 hours of study

Machine learning

Learning from data, and knowing when it worked 50 lessons 1 free
  1. What machine learning is FreePrograms that learn rules from examples instead of being told them.30 min
  2. Features and labelsSupervised learning: inputs, targets and predictions.30 min
  3. Train, validation and test setsHold data back so you can trust your results.38 min
  4. Preparing dataClean, encode and scale data before a model sees it.40 min
  5. Linear regressionFit a line, and predict numbers from features.42 min
  6. Loss functionsOne number that says how wrong the model is.35 min
  7. Training with gradient descentAdjust weights step by step to reduce the loss.42 min
  8. Learning rateStep size, convergence and divergence.35 min
  9. Logistic regressionPredict probabilities for yes/no outcomes.42 min
  10. Multiclass classificationSoftmax and cross-entropy for more than two classes.42 min
  11. The same three models in PyTorchA line, logistic regression and softmax regression fitted both ways: the loss, the gradient and the update opened up, with weight decay and mini-batches.35 min
  12. Precision, recall and F1Accuracy lies; these don't.42 min
  13. ROC curves and AUCJudge a classifier across every threshold.30 min
  14. Calibration: can the probability be trusted?Bands of predicted risk against what happened, the Brier score, why naive Bayes is over-confident, and fixing it on rows the model did not fit.30 min
  15. Overfitting and underfittingMemorizing versus learning.30 min
  16. RegularizationPenalize complexity so models generalize.40 min
  17. Bias and varianceThe two ways a model can be wrong.30 min
  18. Cross-validationReuse limited data for more reliable estimates.42 min
  19. Imbalanced classes in depthRare classes measured: why rebalancing mostly moves the threshold, correcting inflated probabilities, average precision against AUC, why the count of rare cases matters, and thresholds from costs.45 min
  20. Naive Bayes: classifying with countsA classifier trained by counting: Bayes' rule over words, the naive assumption, log scores, smoothing for unseen words, why it is overconfident, and when it beats logistic regression.45 min
  21. Nearest neighborsPredict from the most similar examples.35 min
  22. Decision treesLearn a flowchart of yes/no questions.42 min
  23. Random forests and boostingMany weak models voting beat one strong one.42 min
  24. Gradient boosting in depthBoosting built from scratch: trees fit to the gradient of the loss, depth and interactions, learning rate against number of trees, early stopping, and what XGBoost and LightGBM add.45 min
  25. Support vector machinesThe widest margin, support vectors, the hinge loss and C, kernels for curved boundaries, and when an SVM fits.40 min
  26. Recommender systemsShrunk averages, item-to-item collaborative filtering, matrix factorization, and cold start.40 min
  27. Project: a price modelMultiple regression on log prices, one-hot categories, percentage effects and an extrapolation flag.45 min
  28. Feature engineeringTurn raw data into signals a model can use.35 min
  29. The scikit-learn workflowFit, predict and evaluate with the standard ML library.45 min
  30. Explaining a model: what it uses, and why this predictionPermutation importance on held-back rows, a linear model's contributions for one customer, partial dependence, and what none of them can say.35 min
  31. Capstone: churn predictionPredict which customers will leave, and measure it properly.45 min
  32. ClusteringGroup similar items without labels (k-means).35 min
  33. Clusters of any shape: DBSCAN and hierarchiesWhy k-means needs round clusters, DBSCAN's core points and noise, and hierarchical clustering with its linkages.40 min
  34. Anomaly detectionMasking, robust scores with the median and MAD, baselines from the past, isolation forests and thresholds from a review budget.40 min
  35. Dimensionality reductionVisualize and compress high-dimensional data.35 min
  36. Cleaning messy dataDuplicates, junk rows, text numbers, mixed dates and missing values that mean something.45 min
  37. Grouping, pivoting and reshapingNamed aggregation, pivot tables, transform, running totals and groups too small to rank.45 min
  38. Data in pieces: partitions and mergeable summariesSplit, summarize and combine; which statistics merge exactly, merging variances, sketches for the rest, and partitioning well.30 min
  39. Combining tables with mergeJoin on keys safely, stack tables, and divide by the right denominator.44 min
  40. Time series with pandasResample, smooth, compare across years, and test whether one series leads another.45 min
  41. Forecasting: baselines, lag features and backtestsThree baselines, lag features that never see their own future, a walk-forward backtest, and a range around the number.40 min
  42. Text mining: TF-IDF, search and labelsBag of words and TF-IDF, similarity search and its blind spots, text classifiers and topics, and when LLM labels are worth it.35 min
  43. Location dataHaversine distances, nearest places, GeoJSON, point in polygon and simple maps.40 min
  44. Comparing groups: t-testsWelch's t-test, pooled vs average rates, and a control group for before-and-after changes.40 min
  45. More tests: groups, categories, relationshipsChoose a test, ANOVA by hand, chi-square on counts, Pearson vs Spearman, one-sided tests.40 min
  46. Project: an exploratory analysis reportFive business questions answered with cleaning, joins, charts and a test, then written up.45 min
  47. Bandits: explore or exploitLearning from rewards with no labels: a running estimate for each option, why the greedy choice gets stuck, epsilon-greedy and an optimism bonus.35 min
  48. Decisions in sequence: states, rewards and valueA Markov decision process on a grid world: discounted return, the value of a state, the Bellman equation, and value iteration to the best policy.40 min
  49. Q-learning: learning values by doingA value for every action, learned from experience with no rules given: the update, a grid world you can watch, and why optimism explores better than chance.40 min
  50. Policy gradients: learning what to do directlyA policy as probabilities you can nudge: turn up what worked, measure it against a baseline, and see how the same update trains a language model.40 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write