Q-learning: learning values by doing
A value for every action, learned from experience with no rules given: the update, a grid world you can watch, and why optimism explores better than chance.
What it covers6
Why it matters
Value iteration found the best route through the grid world, and it cheated: it was handed the rules and never took a step. An agent in the real world has no rule book. A program learning a game, a robot learning to walk, a model learning which answers people prefer: each can only try something and see what comes back. Q-learning is the classic method for that. It joins Bandits and Decisions in sequence: the bandit's running estimate, updated from one reward at a time, and the Bellman equation's look ahead to the next state. The result learns the best policy by acting, with no model of the world at all.
What you'll be able to do
- Say what a Q-value is, and why an agent that doesn't know the rules needs one per action
- Apply the Q-learning update to one move: old estimate, plus a step toward reward plus discounted best next value
- Train an agent on a grid world from experience alone and read the policy it learned
- Show a case where random exploring fails, and explain why starting estimates high fixes it
Show the answer
No: values of squares don't help unless it knows which square each move reaches
"Move to the best neighbor" needs to know which move goes to which neighbor, and that is exactly the rule book it lacks. The fix is to learn a value for each action directly: not "how good is this square" but "how good is it to go up from this square". Then choosing is just comparing numbers it has.
The idea: a value for every action
Purpose. Learn how good each move is from experience alone.
The of Decisions in sequence valued states, and needed the rules to turn those values into moves. Without the rules, the agent needs a different table.
is the return the agent can expect if it takes action in state and acts as well as it can from then on: a Q-value, one number for every state and action together. To choose a move, look along the row for the current state and take the action with the largest number.
How does the table get filled? By the bandit's update, with one change. The bandit moved its estimate toward the reward it had just received. Here the reward is only the start of what the move earns, so the agent moves its estimate toward the reward plus what it now believes the next state is worth:
In plain words: the agent was in state , took action , received reward and arrived in state . The target is plus gamma times the largest Q-value in the new state's row, : the Bellman equation, with the agent's own current estimates standing in for the truth. The bracket is the error, target minus old estimate. (alpha) is the learning rate, the fraction of that error to move by.
One update, by hand:
alpha, gamma = 0.5, 0.9 Q = { "A": {"left": 0.0, "right": 0.2}, "B": {"left": 0.1, "right": 0.6}, } # The agent is in A, goes right, gets a reward of -0.04 and finds itself in B. s, a, r, s_next = "A", "right", -0.04, "B" target = r + gamma * max(Q[s_next].values()) print("target:", round(target, 3)) print("error: ", round(target - Q[s][a], 3)) Q[s][a] += alpha * (target - Q[s][a]) print("new Q(A, right):", round(Q[s][a], 3))
The best the agent believes it can do from B is 0.6, so going right from A looks worth −0.04 + 0.9 × 0.6 = 0.5. Its old estimate was 0.2. It moves halfway: 0.35. Nothing else in the table changed.
alpha, gamma = 0.5, 0.9 q_old, reward = 0.4, 1.0 target = reward # the episode ended: nothing follows print(round(q_old + alpha * (target - q_old), 2))
Show the answer
0.7
The episode ended, so the target is the reward alone, 1.0. The error is 1.0 − 0.4 = 0.6, and half of it is 0.3: the estimate moves from 0.4 to 0.7.
One Q-learning update
Write q_update(Q, s, a, reward, s_next, alpha, gamma). Q maps each state to a dictionary of action values. Change Q[s][a] in place: move it by alpha times the error, where the target is reward + gamma * (the largest value in Q[s_next]). If s_next is None, the episode ended and the target is reward alone. Return the error (target minus the old estimate).
def q_update(Q, s, a, reward, s_next, alpha, gamma): return 0.0 Q = {"A": {"left": 0.0, "right": 0.2}, "B": {"left": 0.1, "right": 0.6}} print(round(q_update(Q, "A", "right", -0.04, "B", 0.5, 0.9), 3), Q["A"])
The picture. The update from the cell above, drawn as bars. The estimate Q(A, right) starts at 0.2. The agent's experience gives a target, −0.04 + 0.9 × 0.6 = 0.5. The estimate then moves a fraction alpha = 0.5 of the way toward it.
The same cell prints target 0.5, error 0.3 and new Q 0.35.
Write a sentence or two, then compare it with a good answer in the lesson.
In code: the whole agent, and what it learns
Purpose. Train an agent that knows no rules, compare it with value iteration, and find a case where random exploring fails.
Code flow. The update, line by line, on the numbers from the picture. Step through it and watch the error and the new estimate.
Watching it learn
Here is the whole agent, on the grid world from Decisions in sequence. It chooses moves epsilon-greedily, as the bandit player did, and applies the update after every one. The only thing it ever asks of the world is step: "I did this; what happened?"
import numpy as np ROWS, COLS, WALL, START = 3, 4, (1, 1), (2, 0) ENDS = {(0, 3): 1.0, (1, 3): -1.0} MOVES = {"up": (-1, 0), "down": (1, 0), "left": (0, -1), "right": (0, 1)} ACTIONS = list(MOVES) STATES = [(r, c) for r in range(ROWS) for c in range(COLS) if (r, c) != WALL] ARROW = {"up": "^", "down": "v", "left": "<", "right": ">"} def step(state, action): r, c = state[0] + MOVES[action][0], state[1] + MOVES[action][1] nxt = (r, c) if 0 <= r < ROWS and 0 <= c < COLS and (r, c) != WALL else state return nxt, ENDS.get(nxt, -0.04) def q_learning(episodes, alpha=0.5, gamma=0.9, epsilon=0.1, seed=0): rng = np.random.default_rng(seed) Q = {s: {a: 0.0 for a in ACTIONS} for s in STATES} returns = [] for episode in range(episodes): s, total, moves = START, 0.0, 0 while s not in ENDS and moves < 100: if rng.random() < epsilon: a = ACTIONS[rng.integers(4)] # explore else: a = max(ACTIONS, key=lambda x: Q[s][x]) # exploit nxt, reward = step(s, a) target = reward if nxt in ENDS else reward + gamma * max(Q[nxt].values()) Q[s][a] += alpha * (target - Q[s][a]) s, total, moves = nxt, total + reward, moves + 1 returns.append(total) return Q, returns Q, returns = q_learning(300) for a, b in ((0, 10), (10, 20), (20, 50), (200, 300)): print("episodes", a + 1, "to", b, "| mean return", round(float(np.mean(returns[a:b])), 3)) print("Q at the start square:", {a: round(v, 3) for a, v in Q[START].items()}) for r in range(ROWS): arrows = " ".join( "#" if (r, c) == WALL else "G" if ENDS.get((r, c)) == 1.0 else "P" if (r, c) in ENDS else ARROW[max(ACTIONS, key=lambda a: Q[(r, c)][a])] for c in range(COLS)) values = " ".join( " ## " if (r, c) == WALL else " GOAL " if ENDS.get((r, c)) == 1.0 else " PIT " if (r, c) in ENDS else f"{max(Q[(r, c)].values()):6.3f}" for c in range(COLS)) print(arrows, " | ", values)
The first ten episodes average −0.132: the agent wanders, bumps into walls and sometimes falls in the pit. By episodes 11 to 20 it averages 0.824, close to the 0.84 of the shortest route. It learned the task in about a dozen tries.
Show the answer
Its Q-table: the estimates now point its greedy moves toward the goal
Everything the agent has learned lives in the table. At first every entry is 0 and its greedy choice is no better than chance. After a dozen episodes the entries along the route are large and the entries beside the pit are negative, so the same line of code, "take the action with the largest Q-value", now walks to the goal.
Now compare the grid at the bottom of that output with value iteration's answer from Decisions in sequence. The start square reads 0.519 in both. The squares along the route read 0.621, 0.734 and 0.860 in both. The agent has recovered the exact values, having never seen the rules.
Two squares differ. The middle of the left column reads 0.614 where the true value is 0.621, and the bottom-right corner 0.578 where it is 0.621. The agent rarely goes there, so those estimates have had few updates. Q-learning is only accurate where it has been. That is a weakness and also its strength: it spends its effort on the states that matter for the task, and can work in worlds with far too many states to sweep.
Show the answer
The agent rarely goes there, so its estimate has had few updates
An estimate improves only when the agent takes that action in that state. The route to the goal is traveled in almost every episode, so its values settle. The corner is reached only by an occasional exploring move.
Notice also that the average return near the end, 0.724, is below the 0.84 of a perfect run. The policy it learned is right. But it still explores one move in ten, and now and then a random move drops it in the pit.
Go deeper Learning one policy while following another
The target uses the best next action's value, whatever the agent does next. So Q-learning learns about the greedy policy while behaving epsilon-greedily: off-policy learning. Its cousin SARSA uses the action really taken next, and so learns the value of its own exploring behavior: on-policy. Beside a pit, SARSA learns to keep its distance.
row = {"up": 0.49, "down": 0.423, "left": 0.426, "right": 0.519}
print(max(row, key=row.get), max(row.values()))Show the answer
right 0.519
The greedy policy takes the action with the largest Q-value, here right at 0.519. And the value of a state under that policy is the largest Q-value in its row: V(s) is the maximum of Q(s, a) over the actions.
Show the answer
The rules: where every action leads from every state, and what it pays
Value iteration computes with the rules, calling them for every state and action without moving. Q-learning never sees them. It learns from the moves it actually makes: state, action, reward, next state.
Test what was learned, with exploring off
While learning, the agent's returns are dragged down by its own exploring. To see what it has learned, follow its best moves with no exploring and no updates. Write follow(Q, world, start, max_moves). world(state, action) returns (reward, next_state), with next_state equal to None when the episode ends. From start, keep taking the action with the largest value in Q[state] (the first listed on a tie) until the episode ends or max_moves moves have been made. Return a tuple: the total reward collected, and the number of moves made. Don't change Q.
def follow(Q, world, start, max_moves): return 0.0, 0 def corridor(state, action): if action == "left": return (0.2, None) if state == 0 else (-0.04, state - 1) return (1.0, None) if state == 2 else (-0.04, state + 1) Q = {0: {"left": 0.2, "right": 0.7}, 1: {"left": 0.1, "right": 0.86}, 2: {"left": 0.5, "right": 1.0}} print(follow(Q, corridor, 0, 20))
When random exploring isn't enough
Add one thing to the world: a small prize worth 0.2, in the square right next to the start. Taking it ends the episode. The goal, five moves away, is still worth far more: 0.519 from the start, as you have seen. Will the agent find it?
Try it. The agent below starts with the settings from the cell above.
The widget shows one agent. Here are twenty of each kind, with different seeds, 500 episodes each:
import matplotlib.pyplot as plt import numpy as np ROWS, COLS, WALL, START = 3, 4, (1, 1), (2, 0) # the goal, the pit, and a small prize beside the start ENDS = {(0, 3): 1.0, (1, 3): -1.0, (2, 1): 0.2} MOVES = {"up": (-1, 0), "down": (1, 0), "left": (0, -1), "right": (0, 1)} ACTIONS = list(MOVES) STATES = [(r, c) for r in range(ROWS) for c in range(COLS) if (r, c) != WALL] def step(state, action): r, c = state[0] + MOVES[action][0], state[1] + MOVES[action][1] nxt = (r, c) if 0 <= r < ROWS and 0 <= c < COLS and (r, c) != WALL else state return nxt, ENDS.get(nxt, -0.04) def q_learning(episodes, epsilon, start_value, seed, alpha=0.5, gamma=0.9): rng = np.random.default_rng(seed) Q = {s: {a: start_value for a in ACTIONS} for s in STATES} returns = [] for episode in range(episodes): s, total, moves = START, 0.0, 0 while s not in ENDS and moves < 100: a = (ACTIONS[rng.integers(4)] if rng.random() < epsilon else max(ACTIONS, key=lambda x: Q[s][x])) nxt, reward = step(s, a) target = reward if nxt in ENDS else reward + gamma * max(Q[nxt].values()) Q[s][a] += alpha * (target - Q[s][a]) s, total, moves = nxt, total + reward, moves + 1 returns.append(total) return Q, returns def where_it_ends(Q): s, moves = START, 0 while s not in ENDS and moves < 30: s, moves = step(s, max(ACTIONS, key=lambda a: Q[s][a]))[0], moves + 1 return s for name, epsilon, start_value in (("epsilon 0.1, estimates start at 0", 0.1, 0.0), ("epsilon 0.5, estimates start at 0", 0.5, 0.0), ("epsilon 0, estimates start at 1", 0.0, 1.0)): runs = [q_learning(500, epsilon, start_value, seed) for seed in range(20)] curve = np.mean([returns for Q, returns in runs], axis=0) plt.plot(curve.reshape(-1, 10).mean(axis=1), label=name) found = sum(where_it_ends(Q) == (0, 3) for Q, returns in runs) print(name, "| reaches the goal in", found, "of 20 runs | mean return, last 100 episodes:", round(float(np.mean([np.mean(returns[-100:]) for Q, returns in runs])), 3)) plt.xlabel("episodes, in tens") plt.ylabel("mean return") plt.legend() plt.show()
With epsilon 0.1 and estimates starting at 0, none of the twenty agents learns the way to the goal. All of them take the prize, for a return of about 0.2. Turning exploration up to one move in two only gets 5 of 20 there, and those agents then throw away half their moves.
The third row has no random exploring at all, and all twenty find the goal and collect 0.84.
Here is why. Every estimate starts at 1, higher than any real return in this world. Whatever the agent tries turns out worse than promised, so that estimate drops, and some other, untried action is now the highest in the row. The agent is drawn to whatever it hasn't tried, square after square, until it has tried everything and its estimates are honest. This is the bandit lesson's optimism in the face of uncertainty again, and it explores far more systematically than a coin flip does.
The lesson to take: exploring isn't one dial to turn up. When the reward you want lies many steps beyond a smaller one, how an agent explores matters more than how often.
row = {"up": 1.0, "down": 1.0, "left": 1.0, "right": 1.0}
row["right"] += 0.5 * (0.2 - row["right"])
print(round(row["right"], 2), max(row, key=row.get))Show the answer
0.6 up
The target is 0.2, since the episode ended. The error is 0.2 − 1.0 = −0.8, and half of it takes right down to 0.6. The three untried moves still stand at 1.0, so next time the agent prefers one of those: up is the first.
Worked example: the first two episodes, move by move
The curve above showed the agent going from hopeless to capable in a dozen episodes. Look at how: trace a tiny corridor, where the only reward is at the far end, and watch the reward's news travel back one square per episode.
alpha, gamma = 0.5, 0.9 # squares 0, 1, 2, 3 in a row; square 4 is the goal Q = {square: {"right": 0.0} for square in range(4)} def episode(n): # walk right from square 0 to the goal for s in range(4): reward = 1.0 if s == 3 else 0.0 target = reward if s == 3 else reward + gamma * Q[s + 1]["right"] Q[s]["right"] += alpha * (target - Q[s]["right"]) print("after episode", n, [round(Q[s]["right"], 3) for s in range(4)]) for n in range(1, 7): episode(n) print("true values: ", [round(gamma ** (3 - s), 3) for s in range(4)])
After the first episode only the square beside the goal has learned anything: it received the reward directly. In the second episode the square before it looks ahead, sees a neighbor now worth something, and picks up its first value. Each episode carries the news one square further back, which is why the last line's true values, 0.729, 0.81, 0.9 and 1.0, fill in from the right. This is the same spreading you watched in value iteration's sweeps, but driven by the agent's own walks. It is also why a reward far from the start, like the goal beyond the prize, is so slow to be noticed.
Write a sentence or two, then compare it with a good answer in the lesson.
In practice: when a table is not enough
Purpose. See what replaces the table when states are too many to list, and what to watch in a real system.
What a table can't hold
Q-learning with a table has one hard limit: it needs a row for every state. This world has 11 states. Chess has about 10 to the power 44 positions, far more than any table could hold. A robot's camera image, or the text of a conversation so far, is a state you will never see twice.
The way out is the idea behind every model in this course: don't store an answer for each input, learn a function that produces one. Replace the table with a model that takes a description of the state and outputs a Q-value for each action, and train it so that its outputs move toward the same targets, . States that look alike then share what was learned. When the function is a neural network (Multi-layer networks), this is deep Q-learning, the method that learned to play Atari video games from the screen alone.
The table version is still the right place to have started. The update rule, the target, the trade-off between exploring and exploiting, and the trap of settling for a nearby reward all carry over unchanged.
The line that matters is the loss. The table moved its entry a fraction of the error. The network moves its weights down the gradient of the squared error, which is the same step in a form that generalizes. Deep Q-learning adds a replay buffer of old experience and a slowly-updated copy of the network for the targets, because a target that moves while it is learned is unstable.
Where this turns up in an engineer's week. Game-playing and control agents built on Q-networks; environments with a step function, like the one in the gymnasium library, where env.step(action) returns the next state, a reward and whether the episode ended, exactly the quantities this lesson used; evaluations of agents that act in a loop. The recurring problem is the one in the prize example: an agent that settles for a small reward it found early. When you see an agent plateau, look at how it explores before you change the learning rate.
On the job: run a trained agent with exploring switched off and no updates, as the follow drill did, and report that return. The return while learning understates what it has learned.
Goes deeper in
- Multi-layer networks: the function that replaces the table.
- Policy gradients: learn the policy directly instead of values.
- RLHF: reinforcement learning on a language model.
Write a sentence or two, then compare it with a good answer in the lesson.
Practice what you just read
- 4 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 12 review cards for spaced repetition
Try a quiz question
What does Q(s, a) estimate?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 3 · about 39 hours of study
Machine learning
Learning from data, and knowing when it worked
50 lessons 1 free
- What machine learning is FreePrograms that learn rules from examples instead of being told them.30 min
- Features and labelsSupervised learning: inputs, targets and predictions.30 min
- Train, validation and test setsHold data back so you can trust your results.38 min
- Preparing dataClean, encode and scale data before a model sees it.40 min
- Linear regressionFit a line, and predict numbers from features.42 min
- Loss functionsOne number that says how wrong the model is.35 min
- Training with gradient descentAdjust weights step by step to reduce the loss.42 min
- Learning rateStep size, convergence and divergence.35 min
- Logistic regressionPredict probabilities for yes/no outcomes.42 min
- Multiclass classificationSoftmax and cross-entropy for more than two classes.42 min
- The same three models in PyTorchA line, logistic regression and softmax regression fitted both ways: the loss, the gradient and the update opened up, with weight decay and mini-batches.35 min
- Precision, recall and F1Accuracy lies; these don't.42 min
- ROC curves and AUCJudge a classifier across every threshold.30 min
- Calibration: can the probability be trusted?Bands of predicted risk against what happened, the Brier score, why naive Bayes is over-confident, and fixing it on rows the model did not fit.30 min
- Overfitting and underfittingMemorizing versus learning.30 min
- RegularizationPenalize complexity so models generalize.40 min
- Bias and varianceThe two ways a model can be wrong.30 min
- Cross-validationReuse limited data for more reliable estimates.42 min
- Imbalanced classes in depthRare classes measured: why rebalancing mostly moves the threshold, correcting inflated probabilities, average precision against AUC, why the count of rare cases matters, and thresholds from costs.45 min
- Naive Bayes: classifying with countsA classifier trained by counting: Bayes' rule over words, the naive assumption, log scores, smoothing for unseen words, why it is overconfident, and when it beats logistic regression.45 min
- Nearest neighborsPredict from the most similar examples.35 min
- Decision treesLearn a flowchart of yes/no questions.42 min
- Random forests and boostingMany weak models voting beat one strong one.42 min
- Gradient boosting in depthBoosting built from scratch: trees fit to the gradient of the loss, depth and interactions, learning rate against number of trees, early stopping, and what XGBoost and LightGBM add.45 min
- Support vector machinesThe widest margin, support vectors, the hinge loss and C, kernels for curved boundaries, and when an SVM fits.40 min
- Recommender systemsShrunk averages, item-to-item collaborative filtering, matrix factorization, and cold start.40 min
- Project: a price modelMultiple regression on log prices, one-hot categories, percentage effects and an extrapolation flag.45 min
- Feature engineeringTurn raw data into signals a model can use.35 min
- The scikit-learn workflowFit, predict and evaluate with the standard ML library.45 min
- Explaining a model: what it uses, and why this predictionPermutation importance on held-back rows, a linear model's contributions for one customer, partial dependence, and what none of them can say.35 min
- Capstone: churn predictionPredict which customers will leave, and measure it properly.45 min
- ClusteringGroup similar items without labels (k-means).35 min
- Clusters of any shape: DBSCAN and hierarchiesWhy k-means needs round clusters, DBSCAN's core points and noise, and hierarchical clustering with its linkages.40 min
- Anomaly detectionMasking, robust scores with the median and MAD, baselines from the past, isolation forests and thresholds from a review budget.40 min
- Dimensionality reductionVisualize and compress high-dimensional data.35 min
- Cleaning messy dataDuplicates, junk rows, text numbers, mixed dates and missing values that mean something.45 min
- Grouping, pivoting and reshapingNamed aggregation, pivot tables, transform, running totals and groups too small to rank.45 min
- Data in pieces: partitions and mergeable summariesSplit, summarize and combine; which statistics merge exactly, merging variances, sketches for the rest, and partitioning well.30 min
- Combining tables with mergeJoin on keys safely, stack tables, and divide by the right denominator.44 min
- Time series with pandasResample, smooth, compare across years, and test whether one series leads another.45 min
- Forecasting: baselines, lag features and backtestsThree baselines, lag features that never see their own future, a walk-forward backtest, and a range around the number.40 min
- Text mining: TF-IDF, search and labelsBag of words and TF-IDF, similarity search and its blind spots, text classifiers and topics, and when LLM labels are worth it.35 min
- Location dataHaversine distances, nearest places, GeoJSON, point in polygon and simple maps.40 min
- Comparing groups: t-testsWelch's t-test, pooled vs average rates, and a control group for before-and-after changes.40 min
- More tests: groups, categories, relationshipsChoose a test, ANOVA by hand, chi-square on counts, Pearson vs Spearman, one-sided tests.40 min
- Project: an exploratory analysis reportFive business questions answered with cleaning, joins, charts and a test, then written up.45 min
- Bandits: explore or exploitLearning from rewards with no labels: a running estimate for each option, why the greedy choice gets stuck, epsilon-greedy and an optimism bonus.35 min
- Decisions in sequence: states, rewards and valueA Markov decision process on a grid world: discounted return, the value of a state, the Bellman equation, and value iteration to the best policy.40 min
- Q-learning: learning values by doingA value for every action, learned from experience with no rules given: the update, a grid world you can watch, and why optimism explores better than chance.40 min
- Policy gradients: learning what to do directlyA policy as probabilities you can nudge: turn up what worked, measure it against a baseline, and see how the same update trains a language model.40 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write