Training reasoning models

Watching GRPO learn addition

Below, a small network learns to add two single digits, such as 3 + 5. It learns from the checker alone. It never sees a correct answer. It only finds out whether each of its own sampled answers was right.

Here's what the code does:

1. Each problem is two digits. The encode function turns each digit into a one-hot list: ten numbers, all 0 except a 1 in that digit's spot.
2. The policy is a small neural network. For each problem, it gives a chance to each of the 19 possible answers, 0 through 18.
3. Each training step picks 32 random problems. For each one, it samples a group of answers using those chances.
4. The checker gives 1 for right and 0 for wrong. Each answer then gets its advantage within its group.
5. The loss raises the chances of answers with a positive advantage. It lowers the chances of answers with a negative advantage.
6. Every 200 steps, it tests the policy's top answer on all 100 problems, from 0 + 0 to 9 + 9.

It runs twice: first with groups of 2, then with groups of 16. It's slow, so the Run button counts the seconds.

import torch
import torch.nn as nn
import torch.nn.functional as F

def encode(a, b):
    return torch.cat([F.one_hot(a, 10), F.one_hot(b, 10)], dim=1).float()

all_a, all_b = torch.arange(100) // 10, torch.arange(100) % 10      # every problem from 0+0 to 9+9

def train(group_size, steps=801):
    torch.manual_seed(0)
    policy = nn.Sequential(nn.Linear(20, 64), nn.ReLU(), nn.Linear(64, 19))   # answers 0..18
    opt = torch.optim.Adam(policy.parameters(), lr=3e-3)
    for step in range(steps):
        a, b = torch.randint(0, 10, (32,)), torch.randint(0, 10, (32,))
        logits = policy(encode(a, b))
        answers = torch.multinomial(torch.softmax(logits, dim=-1), group_size, replacement=True)
        reward = (answers == (a + b).unsqueeze(1)).float()                # the checker: right or wrong
        advantage = (reward - reward.mean(dim=1, keepdim=True)) / (reward.std(dim=1, keepdim=True) + 1e-4)
        logp = torch.log_softmax(logits, dim=-1).gather(1, answers)
        loss = -(advantage * logp).mean()
        opt.zero_grad()
        loss.backward()
        opt.step()
        if step % 200 == 0:
            with torch.no_grad():
                accuracy = (policy(encode(all_a, all_b)).argmax(dim=-1) == all_a + all_b).float().mean().item()
            print(f"  group of {group_size:>2}, step {step:>3}: {accuracy:.0%} of all 100 problems right")

train(2)
train(16)

Both runs start near chance, about 1 right in 19. With groups of 16, the policy gets 95% of all problems right within 400 steps. With groups of 2, it learns far more slowly. After 800 steps it reaches only 66%.

The reason is worth remembering. When every answer in a group gets the same reward, every advantage is zero. Both wrong, or both right, and the problem teaches nothing. Bigger groups make it likelier that at least one attempt succeeds. So even hard problems give a learning signal.

The 16-group run's last 5% stalls for the same reason. On those problems, the policy has settled on a wrong answer. It almost never samples the right one, so it has no right answer to learn from.

That's why real training tunes how hard the problems are. A problem the model always solves, or never solves, is wasted compute. (torch here is the course's in-browser PyTorch subset.)

p_wrong = 0.9
print(round(p_wrong ** 2, 2), round(p_wrong ** 16, 2))

Checkers can be gamed too

A verifiable reward is only as good as its checker. Picture a student who memorizes the answers to the practice test. They ace it, but they can't solve a new problem.

If the tests are weak, a policy can learn to pass them without solving the problem. RL is very good at finding shortcuts like that. Labs have reported coding models trained this way that special-case the tests, writing code that only handles the test inputs. Others edited the tests instead of fixing the code.

The code below has two "solutions" to adding two numbers. honest adds them. hack memorizes the two test cases and returns 0 for anything else. Two checkers judge them. weak_tests tries the two known cases. strong_tests tries 100 random pairs of numbers between −99 and 99.

import random

def honest(a, b):
    return a + b

def hack(a, b):                                  # memorizes the visible test cases
    return {(2, 3): 5, (10, 4): 14}.get((a, b), 0)

def weak_tests(f):
    return f(2, 3) == 5 and f(10, 4) == 14

def strong_tests(f):
    random.seed(0)
    return all(f(a, b) == a + b for a, b in [(random.randint(-99, 99), random.randint(-99, 99)) for _ in range(100)])

for f in (honest, hack):
    print(f"{f.__name__:<7} weak tests pass: {weak_tests(f)!s:<5}  strong tests pass: {strong_tests(f)}")

Both "solutions" earn full reward from the weak tests. Only the honest one passes the strong tests. Defenses include:

• hidden and randomized tests;
• many varied problems;
• checks for suspicious edits;
• monitoring what the model writes in its reasoning.

def hack(a, b):
    return {(2, 3): 5, (10, 4): 14}.get((a, b), 0)

print(hack(7, 8))

Thinking more at answer time

Training is one place to spend compute. Answering is another. Reasoning models spend it by thinking longer. That's the "effort" or "thinking budget" setting from Phase 1.

Two simpler tricks work with any model:

• Majority vote, also called self-consistency: sample several answers and go with the most common one. It's like asking a room full of people and going with the most popular reply.
• pass@k (say "pass at k"): make several attempts and keep any answer a checker accepts. This needs a checker, such as tests for code.

The chance that at least one attempt is right

Say each attempt is right with chance $p$, and the attempts don't affect each other. The chance that at least one of $k$ attempts is right is:

Here $p$ is the chance that one attempt is right, and $k$ is the number of attempts.

In plain words: $1 - p$ is the chance one attempt is wrong. Multiply $k$ copies of it together to get the chance that every attempt is wrong. Subtract that from 1.

For example, say each attempt is right 40% of the time, so $p = 0.4$. One attempt is wrong with chance 0.6. All five of five attempts are wrong with chance 0.6 × 0.6 × 0.6 × 0.6 × 0.6, about 0.078. So at least one is right with chance 1 − 0.078 = 0.922, about 92%.

p, k = 0.4, 5
all_wrong = (1 - p) ** k
print(round(all_wrong, 3), round(1 - all_wrong, 3))

Real attempts from one model aren't fully independent, so treat this as a best case.

Majority vote

Majority vote needs no checker. It needs something else: wrong answers that disagree with each other. Say five attempts at 17 + 25 give 42, 42, 41, 43 and 32. The right answer, 42, wins with only two votes, because the three wrong answers split.

The simulation below tests both tricks. Each attempt is right 40% of the time. A wrong attempt lands on one of 6 different wrong answers, at random. It runs 2,000 trials for each number of attempts.

import numpy as np

rng = np.random.default_rng(0)
p_right, n_wrong_answers = 0.4, 6           # each attempt is right 40% of the time; wrong ones scatter

def vote(samples):
    values, counts = np.unique(samples, return_counts=True)
    return rng.choice(values[counts == counts.max()])      # break ties at random

for n in [1, 5, 15, 45]:
    trials, by_vote, any_right = 2000, 0, 0
    for _ in range(trials):
        samples = np.where(rng.random(n) < p_right, -1, rng.integers(0, n_wrong_answers, n))   # -1 = correct
        by_vote += vote(samples) == -1
        any_right += (samples == -1).any()
    print(f"{n:>2} attempts: majority vote right {by_vote / trials:.0%}   at least one right (pass@{n}) {any_right / trials:.0%}")

With one attempt, both measures sit near 40%. As attempts grow, pass@k climbs fastest: about 92% at 5 attempts, as the formula predicted. Majority vote climbs more slowly, but it reaches about 99% at 45 attempts.

So a model that's right only 40% of the time per attempt becomes very reliable with enough votes. That holds as long as its wrong answers disagree with each other. This is the logic of test-time compute: you can buy accuracy at answer time, paid for in tokens and waiting time. It explains why reasoning models get more accurate as their thinking budget grows. It's also why the best results on hard benchmarks often use a lot of compute per problem.

What's still hard

• Fuzzy tasks. RLVR shines where answers can be checked. Writing, strategy and advice have no checker. Those still lean on preference training and AI feedback.
• Faithfulness. A model's written reasoning isn't guaranteed to be the real reason for its answer. In 2025, Anthropic researchers found that reasoning models often didn't mention hints they had used. Treat a chain of thought as useful evidence, not a guaranteed record.
• Overthinking. Long reasoning costs tokens and time, even on easy questions. Choosing the effort level for each task is part of using these models well.

For marketers: for analytical tasks, such as attribution math, budget allocation or cleaning messy data, use a reasoning model or higher effort. For brainstorming taglines, a fast model sampled several times usually serves better than long deliberation.