Capstone: a tuned, evaluated assistant

How the test works

Every number comes from 72 product prompts. They use the six test names from the fine-tune project, and those names were never used for training or for choosing settings. The model answers each prompt three times, which makes 216 answers.

Why three answers each? A model's answers vary from run to run. With one answer per prompt, chance alone could move a guideline's rate by more than gate 2's 5-point margin. You did this arithmetic in the benchmarks lesson.

import math
p = 0.5                          # a guideline broken in half the answers: the noisiest case
for n in (72, 216):
    se = math.sqrt(p * (1 - p) / n)
    print(f"{n} answers: about {se * 100:.1f} points")

Candidate A: curated model outputs

The first idea reuses the fine-tune project. It runs SFT (supervised fine-tuning, training on example answers) for 80 steps. The examples are the chosen answers that passed every guideline check.

The cell below does the whole review. It trains the candidate. It tests the untouched baseline and the candidate on the same 72 prompts, three answers each. Then it applies the three gates. It takes a while, and the Run button counts the seconds. (torch here is the course's in-browser PyTorch subset.)

import requests
import torch
import torch.nn as nn
from tinygpt import load_pretrained

baseline, tok = load_pretrained("tiny-gpt-chat")
candidate, _ = load_pretrained("tiny-gpt-chat")
end = tok.token_id("<end>")
pairs = requests.get("https://api.gradient.test/v1/preferences").json()["data"]
kinds = sorted({" ".join(p["prompt"].split()[3:-1]) for p in pairs})
test_names = {"Kestrel", "Lark", "River", "Sierra", "Summit", "Willow"}
held_back = test_names | {"Alder", "Basin", "Cove", "Ember"}        # test and validation names

def to_tensors(rows):
    enc = []
    for prompt, answer in rows:
        p, a = tok.encode(f"<user> {prompt} <assistant>"), tok.encode(answer) + [end]
        enc.append(((p + a)[:65], ([-100] * len(p) + a)[:65]))
    T = max(len(ids) for ids, _ in enc)
    return (torch.tensor([ids + [end] * (T - len(ids)) for ids, _ in enc]),
            torch.tensor([lab + [-100] * (T - len(lab)) for _, lab in enc]))

def finetune(model, rows, steps=80):
    X, Y = to_tensors(rows)
    opt = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)
    model.train()
    torch.manual_seed(1)
    for step in range(steps):
        b = torch.randint(0, len(X), (16,))
        logits = model(X[b][:, :-1])
        loss = nn.functional.cross_entropy(logits.reshape(-1, logits.shape[-1]), Y[b][:, 1:].reshape(-1))
        opt.zero_grad()
        loss.backward()
        opt.step()
    model.eval()

@torch.no_grad()
def evaluate(model):
    torch.manual_seed(0)
    items, clean = [], 0
    for product in [f"{n} {k}" for n in sorted(test_names) for k in kinds]:
        ids = torch.tensor([tok.encode(f"<user> Describe the {product} . <assistant>")] * 3)   # three answers each
        for row in model.generate(ids, 60, temperature=0.8, stop_id=end)[:, ids.shape[1]:].tolist():
            reply = tok.decode(row)
            clean += reply.endswith("<end>")
            items.append({"product": product, "text": reply.split("<end>")[0].strip()})
    flaws = [r["flaws"] for r in requests.post("https://api.gradient.test/v1/brand-check", json={"items": items}).json()["results"]]
    share = lambda test: sum(any(test(f) for f in fs) for fs in flaws) / len(flaws)
    return {"violations per answer": sum(map(len, flaws)) / len(flaws),
            "names another product": share(lambda f: f.startswith("doesn't name")),
            "repeats a word": share(lambda f: f.startswith("repeats")),
            "wrong or missing material": share(lambda f: "material" in f or "isn't used" in f),
            "missing repair promise": share(lambda f: f == "no repair promise"),
            "ends cleanly": clean / len(items)}

def gates(before, after):
    guidelines = ["names another product", "repeats a word", "wrong or missing material", "missing repair promise"]
    return {"violations down at least 20%": after["violations per answer"] <= 0.8 * before["violations per answer"],
            "no guideline worse by > 5 points": all(after[g] - before[g] <= 0.05 for g in guidelines),
            "at least 90% end cleanly": after["ends cleanly"] >= 0.9}

curated = [(p["prompt"], p["chosen"]) for p in pairs if not p["chosen_flaws"] and p["prompt"].split()[2] not in held_back]
finetune(candidate, curated)
before, after = evaluate(baseline), evaluate(candidate)
print(f"candidate A: {len(curated)} curated answers\n")
print(f"{'':<28} {'baseline':>9} {'candidate':>10}")
for metric in before:
    print(f"{metric:<28} {before[metric]:>9.2f} {after[metric]:>10.2f}")
print()
for gate, ok in gates(before, after).items():
    print(f"{'PASS' if ok else 'FAIL'}  {gate}")

Reading the result

The headline metric is the one number that sums up the result: violations per answer. It looks like a success. It fell from 1.57 to 1.11. Missing repair promises fell from 77% to 25%.

But the second gate fails. Answers that name the wrong product rose from 42% to 51%. The model learned the repair promise. It also got worse at saying which product it's describing. A team that watched only the headline metric would have shipped it.

before, after = 1.57, 1.11                # violations per answer
print(f"gate 1 target: {0.8 * before:.2f} or fewer")
print(f"cut: {(before - after) / before:.0%}")

rise = 0.51 - 0.42                         # wrong-product answers, as shares
print(f"wrong product: up {rise * 100:.0f} points")
print("gate 2:", "PASS" if rise <= 0.05 else "FAIL")

Diagnosis

Why did the model get worse at naming the product? Look at the training data.

The 61 curated answers are the chat model's own outputs. They're only the answers that passed the checker, and those lean toward a narrow slice of products and phrasings. Training on them strengthens the model's habits. One of those habits is drifting toward familiar products. So the brand rule was learned, but the facts had nothing to hold them in place.

Think of a cook who learns only from photos of their own best dishes. They get better at plating. They also repeat their old mistakes, because nothing new is coming in.

The fix is in the data. The catalog's real descriptions are right about the facts by definition. Add the brand rule to them: append "Backed by our repair promise." where it's missing. Then train on those. The model gets real facts plus the guideline.

Candidate B: real facts plus the rule

Candidate B keeps the same training settings, the same test and the same gates. Only the data changes. The cell reads each catalog description and picks out the product's name and kind. It skips the held-back names. It adds the repair promise where it's missing. Then it trains and runs the review.

import re
import requests
import torch
import torch.nn as nn
from tinygpt import load_pretrained

baseline, tok = load_pretrained("tiny-gpt-chat")
candidate, _ = load_pretrained("tiny-gpt-chat")
end = tok.token_id("<end>")
pairs = requests.get("https://api.gradient.test/v1/preferences").json()["data"]
docs = requests.get("https://api.gradient.test/v1/corpus").json()["text"].strip().split("\n")
kinds = sorted({" ".join(p["prompt"].split()[3:-1]) for p in pairs})
test_names = {"Kestrel", "Lark", "River", "Sierra", "Summit", "Willow"}
held_back = test_names | {"Alder", "Basin", "Cove", "Ember"}

def to_tensors(rows):
    enc = []
    for prompt, answer in rows:
        p, a = tok.encode(f"<user> {prompt} <assistant>"), tok.encode(answer) + [end]
        enc.append(((p + a)[:65], ([-100] * len(p) + a)[:65]))
    T = max(len(ids) for ids, _ in enc)
    return (torch.tensor([ids + [end] * (T - len(ids)) for ids, _ in enc]),
            torch.tensor([lab + [-100] * (T - len(lab)) for _, lab in enc]))

def finetune(model, rows, steps=80):
    X, Y = to_tensors(rows)
    opt = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)
    model.train()
    torch.manual_seed(1)
    for step in range(steps):
        b = torch.randint(0, len(X), (16,))
        logits = model(X[b][:, :-1])
        loss = nn.functional.cross_entropy(logits.reshape(-1, logits.shape[-1]), Y[b][:, 1:].reshape(-1))
        opt.zero_grad()
        loss.backward()
        opt.step()
    model.eval()

@torch.no_grad()
def evaluate(model):
    torch.manual_seed(0)
    items, clean = [], 0
    for product in [f"{n} {k}" for n in sorted(test_names) for k in kinds]:
        ids = torch.tensor([tok.encode(f"<user> Describe the {product} . <assistant>")] * 3)   # three answers each
        for row in model.generate(ids, 60, temperature=0.8, stop_id=end)[:, ids.shape[1]:].tolist():
            reply = tok.decode(row)
            clean += reply.endswith("<end>")
            items.append({"product": product, "text": reply.split("<end>")[0].strip()})
    flaws = [r["flaws"] for r in requests.post("https://api.gradient.test/v1/brand-check", json={"items": items}).json()["results"]]
    share = lambda test: sum(any(test(f) for f in fs) for fs in flaws) / len(flaws)
    return {"violations per answer": sum(map(len, flaws)) / len(flaws),
            "names another product": share(lambda f: f.startswith("doesn't name")),
            "repeats a word": share(lambda f: f.startswith("repeats")),
            "wrong or missing material": share(lambda f: "material" in f or "isn't used" in f),
            "missing repair promise": share(lambda f: f == "no repair promise"),
            "ends cleanly": clean / len(items)}

def gates(before, after):
    guidelines = ["names another product", "repeats a word", "wrong or missing material", "missing repair promise"]
    return {"violations down at least 20%": after["violations per answer"] <= 0.8 * before["violations per answer"],
            "no guideline worse by > 5 points": all(after[g] - before[g] <= 0.05 for g in guidelines),
            "at least 90% end cleanly": after["ends cleanly"] >= 0.9}

product = re.compile(r"^(?:Meet the|Our|The) ([A-Z][a-z]+) (.+?)(?:,| is| are) made")
rows = []
for d in docs:                                         # the catalog's real descriptions...
    name, kind = product.match(d).groups()
    if name in held_back:
        continue
    answer = d if "repair promise" in d else d + " Backed by our repair promise."   # ...plus the brand rule
    rows.append((f"Describe the {name} {kind} .", answer))

finetune(candidate, rows)
before, after = evaluate(baseline), evaluate(candidate)
print(f"candidate B: {len(rows)} catalog descriptions with the repair promise\n")
print(f"{'':<28} {'baseline':>9} {'candidate':>10}")
for metric in before:
    print(f"{metric:<28} {before[metric]:>9.2f} {after[metric]:>10.2f}")
print()
results = gates(before, after)
for gate, ok in results.items():
    print(f"{'PASS' if ok else 'FAIL'}  {gate}")
print("\nDecision:", "ship candidate B" if all(results.values()) else "do not ship")

Reading the result

Candidate B passes every gate.

• Violations per answer fall by about 39%, from 1.57 to 0.96.
• Missing repair promises fall from 77% to 8%.
• Every other guideline stays within the agreed margin. Naming the wrong product and repeating words each rise 4 points. Materials move 1 point.

Same model, same training settings, same number of steps. Better data made the difference. That's the usual story in fine-tuning.

What a raw loss would have missed

One more lesson hides in what the team didn't gate on: the loss. Cross-entropy measures how surprised a model is by some text. Lower means less surprised.

In testing, both candidates' cross-entropy on the catalog's original descriptions went up. B's went up more than A's. Part of the reason is that B now always adds a sentence those descriptions lack. So the loss rose partly because B learned the intended change.

A raw loss can't tell "learned the intended change" from "got worse". Checks tied to what you actually care about can.

Ship it, with a model card

A model card is a short record that travels with a model. It says what the model is for, how it was made, how it was tested and where it falls short. Margaret Mitchell and colleagues proposed model cards in 2019. They're now standard on the Hugging Face Hub. Here's the card for candidate B:

{
  "model": "tiny-gpt-brand-v1",
  "base_model": "tiny-gpt-chat",
  "intended_use": "First drafts of product descriptions in the brand voice, reviewed by an editor before publishing",
  "not_for": "Facts about products missing from the catalog; customer-facing use without review",
  "training": "SFT, 80 steps, lr 3e-4, on 193 real catalog descriptions with the repair promise added",
  "evaluation": "216 answers to 72 held-out products, brand-check API; violations per answer 1.57 -> 0.96; missing repair promise 77% -> 8%",
  "release_gates": "all passed: violations -39%, no guideline worse by more than 5 points, 94% end cleanly",
  "known_limitations": "Still names the wrong product in nearly half of answers; a tiny teaching model",
  "date": "2026-10"
}

Read it like a label on a food package:

• intended_use and not_for say where the model belongs, and where it doesn't.
• training and evaluation say how it was made and how it scored, with real numbers.
• release_gates records that every gate passed.
• known_limitations is honest about the weak spots. B still names the wrong product in nearly half of answers.

Shipping isn't the end of the work. After release, the team monitors a sample of real outputs with the same checks. It adds new failure cases to the test set. And it reruns the gates for every new version.

Where to go from here

• Do it for real. Repeat this workflow on a small open model. Use the TRL and PEFT code from the fine-tune project and LoRA lessons. Bring your own data, your own checker or a validated LLM judge, and gates agreed in advance.
• Add preference training. Run DPO on top of candidate B. Use pairs that differ on the weakest guideline: naming the right product, still wrong in nearly half of answers. Then check the gates again.
• Keep the habits. Choose the metric first. Test on held-out data. Check every criterion. Set gates before you see results. Fix the data before you reach for tricks. And write it all down.

For marketers: you can now run a model customization the way a careful AI team would, or commission one and review it. Ask for four things: the brief, the gates, the per-guideline table and the model card. If any of the four is missing, the project isn't finished.