DPO and direct preference methods

import math

beta = 0.5
ref_chosen, ref_rejected = -10.0, -12.0     # reference log-probabilities
pol_chosen, pol_rejected = -9.0, -13.0      # policy log-probabilities, after some training

margin = (pol_chosen - ref_chosen) - (pol_rejected - ref_rejected)
sigma = 1 / (1 + math.exp(-beta * margin))
loss = -math.log(sigma)
print(margin, round(sigma, 3), round(loss, 3))

A toy run: five possible answers

Here is DPO on the smallest model possible. The "policy" is nothing but a chance for each of five possible answers, and it starts equal to the reference. Simulated raters judge 300 pairs by a hidden quality score. The bigger the quality gap in a pair, the more often the better answer wins. DPO only ever sees who won.

• p_ref holds the reference's chance for each answer.
• quality is the hidden score the raters judge by.
• The first loop builds the pairs: winners go in W, losers in L.
• The training loop computes every pair's margin and the DPO loss, then takes 200 small steps.

import numpy as np
import torch
import torch.nn.functional as F

answers = ["on brand, clean", "on brand, plain", "repeats a word", "wrong material", "no repair promise"]
p_ref = torch.tensor([0.30, 0.35, 0.15, 0.15, 0.05])
quality = np.array([1.0, 0.7, -0.5, -0.8, -0.3])              # hidden; raters judge by it

rng = np.random.default_rng(1)
pairs = []
for _ in range(300):
    i, j = rng.choice(5, size=2, replace=False)
    i_wins = rng.random() < 1 / (1 + np.exp(-(quality[i] - quality[j])))
    pairs.append((i, j) if i_wins else (j, i))
W, L = torch.tensor(pairs).T                                   # winners, losers

beta = 0.5
logits = torch.log(p_ref).clone().requires_grad_(True)         # start the policy at the reference
opt = torch.optim.Adam([logits], lr=0.05)
for step in range(200):
    logp, logp_ref = torch.log_softmax(logits, dim=0), torch.log(p_ref)
    margin = (logp[W] - logp_ref[W]) - (logp[L] - logp_ref[L])
    loss = -F.logsigmoid(beta * margin).mean()
    opt.zero_grad()
    loss.backward()
    opt.step()

with torch.no_grad():
    p = torch.softmax(logits, dim=0)
    implicit = beta * (torch.log(p) - torch.log(p_ref))
print(f"{'answer':<20} {'reference':>9} {'after DPO':>9} {'implicit reward':>15}")
for a, r0, r1, ir in zip(answers, p_ref.tolist(), p.tolist(), implicit.tolist()):
    print(f"{a:<20} {r0:>9.2f} {r1:>9.2f} {ir:>15.2f}")

Probability moves toward the answers the raters preferred. The two on-brand answers end up with nearly all of it. The implicit rewards in the last column come out in the same order as the hidden quality, though DPO never saw a single score. (torch here is the course's in-browser PyTorch subset.)

DPO on a real model

Now for DPO on tiny-gpt-chat, with the 239 training pairs from the last lesson's brand dataset. Recall the brand reviewer's four guidelines. Name the product that was asked about. Don't repeat a word in a sentence. Use a material the catalog uses for that kind of product. Mention the repair promise.

A whole answer needs a sequence log-probability. That's the sum of the log-probabilities the model gives each of the answer's tokens, one after another. Only the answer's tokens count, not the prompt's: the same masking as SFT. In the code, logprob does this for a batch of answers.

The reference model's values are worked out once, since it never changes. The policy's are worked out again every step. Training uses $\beta = 0.1$ and batches of 16 pairs. Afterwards, both models describe products held out of training. The brand reviewer then counts how often each guideline is broken. It takes about half a minute.

import requests
import torch
import torch.nn.functional as F
from tinygpt import load_pretrained

policy, tok = load_pretrained("tiny-gpt-chat")       # the model we train
reference, _ = load_pretrained("tiny-gpt-chat")      # a frozen copy of where it started
end = tok.token_id("<end>")
pairs = requests.get("https://api.gradient.test/v1/preferences").json()["data"]
held_out = {"Kestrel", "Lark", "River", "Sierra", "Summit", "Willow"}
train = [p for p in pairs if p["prompt"].split()[2] not in held_out]

def encode(prompt, answer):
    p, a = tok.encode(f"<user> {prompt} <assistant>"), tok.encode(answer) + [end]
    return (p + a)[:65], ([-100] * len(p) + a)[:65]

def logprob(model, rows):
    # Total log-probability the model gives each answer, given its prompt.
    T = max(len(ids) for ids, _ in rows)
    ids = torch.tensor([ids + [end] * (T - len(ids)) for ids, _ in rows])
    labels = torch.tensor([lab + [-100] * (T - len(lab)) for _, lab in rows])[:, 1:]
    logp = torch.log_softmax(model(ids[:, :-1]), dim=-1)
    picked = logp.gather(-1, labels.clamp(min=0).unsqueeze(-1)).squeeze(-1)
    return (picked * (labels != -100).float()).sum(dim=-1)

chosen = [encode(p["prompt"], p["chosen"]) for p in train]
rejected = [encode(p["prompt"], p["rejected"]) for p in train]
with torch.no_grad():
    ref_chosen, ref_rejected = logprob(reference, chosen), logprob(reference, rejected)

kinds = sorted({" ".join(p["prompt"].split()[3:-1]) for p in pairs})
products = [f"{n} {k}" for n in sorted(held_out) for k in kinds]

@torch.no_grad()
def violations(model):
    items = []
    for product in products:
        ids = torch.tensor([tok.encode(f"<user> Describe the {product} . <assistant>")] * 2)
        for row in model.generate(ids, 60, temperature=0.8, stop_id=end)[:, ids.shape[1]:].tolist():
            items.append({"product": product, "text": tok.decode(row).split("<end>")[0].strip()})
    flaws = [r["flaws"] for r in requests.post("https://api.gradient.test/v1/brand-check", json={"items": items}).json()["results"]]
    rate = lambda test: sum(any(test(f) for f in fs) for fs in flaws) / len(flaws)
    return {"names another product": rate(lambda f: f.startswith("doesn't name")),
            "repeats a word": rate(lambda f: f.startswith("repeats")),
            "wrong or no material": rate(lambda f: "material" in f or "isn't used" in f),
            "no repair promise": rate(lambda f: f == "no repair promise"),
            "any violation": sum(bool(fs) for fs in flaws) / len(flaws)}

torch.manual_seed(0)
before = violations(policy)

beta = 0.1
opt = torch.optim.Adam(policy.parameters(), lr=2e-4)
policy.train()
torch.manual_seed(1)
for step in range(61):
    b = torch.randint(0, len(train), (16,)).tolist()
    margin = (logprob(policy, [chosen[i] for i in b]) - ref_chosen[b]) - (logprob(policy, [rejected[i] for i in b]) - ref_rejected[b])
    loss = -F.logsigmoid(beta * margin).mean()
    opt.zero_grad()
    loss.backward()
    opt.step()
    if step % 20 == 0:
        print(f"step {step:>2}  loss {loss.item():.3f}  chosen ahead in {(margin > 0).float().mean().item():.0%} of the batch")

policy.eval()
torch.manual_seed(0)
after = violations(policy)
print(f"\n{'share of answers that...':<24} {'before':>7} {'after':>7}")
for rule in before:
    print(f"{rule:<24} {before[rule]:>7.0%} {after[rule]:>7.0%}")

Read the table carefully, because it's typical. DPO learned one guideline decisively. Answers without the repair promise fell from 81% to 19%, on products it was never trained on. The other three didn't improve: all three got slightly worse. There are three reasons.

• DPO learns whatever separates the pairs most easily. The repair promise is a fixed phrase. In this dataset, it almost only appears on the chosen side: 130 chosen answers have it, and only 14 rejected ones. That's a clean, steady signal. Naming the right product is harder. The model has to copy a name it may rarely have seen in that spot.
• Preference training reshuffles habits the model already has. It can turn a phrase the model sometimes writes into one it nearly always writes. It can't add a skill the model lacks.
• Side effects are normal. Pushing probability toward chosen answers moves it away from other things too. So always test every rule you care about, not only the one you aimed at. Treat a regression like this as a side effect to evaluate on every criterion: adjust the data, the beta or the training length, and keep the change only if the trade-off is acceptable. Push harder and it gets dramatic. In testing, 100 steps at ten times this learning rate made the model mention the repair promise almost every time. But 96% of its answers named the wrong product, and 99% broke at least one guideline.

DPO in practice

Why it caught on. DPO trains like SFT. You need a fixed dataset, a loss and two models: the policy and a frozen reference. Nothing is generated during training. It's far cheaper and more stable than PPO. Open models such as Zephyr, Tulu 3 and Meta's Llama 3 used it, or close variants, in post-training.

PPO-style RL still has an edge in one case: when you can generate fresh answers and score them during training. That's called online training. The model learns from its own current mistakes, not from a fixed set of pairs. Some teams get part of that benefit by running DPO in rounds, making new pairs each time.

The knobs. $\beta$ is often 0.1. It sets how far the policy may move from the reference. Smaller values let it chase the preferences harder. Learning rates are small: around $10^{-6}$, one millionth, for large models. One to three passes over the data are typical.

Variants you'll see:

• IPO copes better with noisy preferences, or ones where the same answer nearly always wins.
• KTO learns from single thumbs-up or thumbs-down labels instead of pairs.
• ORPO and SimPO drop the reference model.

Several also add an SFT term on the chosen answers, to keep the policy fluent.

With Hugging Face's TRL library, the dataset has prompt, chosen and rejected columns, like the practice API's:

from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import DPOConfig, DPOTrainer

name = "Qwen/Qwen2.5-0.5B-Instruct"
model, tokenizer = AutoModelForCausalLM.from_pretrained(name), AutoTokenizer.from_pretrained(name)
dataset = load_dataset("json", data_files="brand_preferences.jsonl", split="train")
trainer = DPOTrainer(model=model, args=DPOConfig(output_dir="brand-dpo", beta=0.1),
                     train_dataset=dataset, processing_class=tokenizer)
trainer.train()        # the frozen reference copy is created for you

For marketers: preference tuning is good at building steady, recognizable habits. Always mention the guarantee, end with a call to action, keep it under 50 words. It won't make a model know your product line. That still needs your data in the prompt.