Index Ai Academy
Phase 6 · Why models do that

Why it never says "I don't know"

A chat model answers every request because its fine-tuning data always did; preference training adds flattery; how to build abstention in.

About 25 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers9

Why it matters

Ask an assistant about a paper that was never written and it summarizes it. Tell it your wrong answer is right and it may agree. The confident-and-wrong lesson explained why a base model cannot say "I don't know". This lesson explains why a chat model, which could have been taught to, usually does not: its fine-tuning data taught it that every request gets a fluent answer, and preference training then rewarded the answers people liked, the confident and agreeable ones.

The course's two tiny models make this concrete: they share every weight except what 600 steps of fine-tuning on 300 "describe this product" chats changed, and that is enough to produce "The Cove tent is made from silnylon." for a product that was never in the catalog. Seeing the mechanism tells you what to change: the data, the output format and the evaluation, not the prompt's tone.

What you'll be able to do

  • Show that a chat-tuned model answers every request, including ones about things that do not exist
  • Trace the behavior to the fine-tuning data: every example answers, none declines
  • Explain how preference training extends compliance into flattery
  • Build abstention in: data, output schemas, and an evaluation that rewards declining
Quick check
A chat model was fine-tuned on 300 conversations in which a user asks for a product description and the assistant writes one. It is asked to describe a product that does not exist. A guess at what it does:
  1. Returns the unknown token and stops, since the name is outside its vocabulary
  2. Writes a fluent description anyway, every time, borrowing a real product's details
  3. Says it has no information about that product and asks for another
  4. Asks the user which product they mean, as assistants are trained to do
Show the answer

Writes a fluent description anyway, every time, borrowing a real product's details

Every example it learned from answers. Nothing in its training showed what declining looks like, so the most likely continuation after the assistant's turn begins is the start of a description, whatever the request named.

The idea: every training reply starts with an answer

Purpose. Why a chat model answers every request, traced to what its fine-tuning data contained.

This lesson rests on Supervised fine-tuning, where a model is fine-tuned on example chats, and on Confident and wrong.

The course has two tiny GPTs. tiny-gpt is the base model: 2 layers, 40,195 parameters, trained to predict the next token of 300 product descriptions. tiny-gpt-chat started from those exact weights and took 600 steps of supervised fine-tuning (Supervised fine-tuning) on chats of one shape: <user> Describe the Harbor camp mug . <assistant> followed by the catalog's description of the Harbor camp mug and an <end> token. Three hundred such chats, one per product, with the loss computed on the assistant's part only.

That is the whole difference between the two models. Whatever the chat model does that the base model does not, those 600 steps on those 300 examples taught it. The next cell gives both models a request about a product that is not in the catalog, the Zephyr tent, and prints what each writes.

Worked example

A request about a product that does not exist

The same prompt to both models, 25 tokens, temperature 0.7, seed 0. The chat model stops at <end>.

Setup (runs before every cell) · 5 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import torch
from tinygpt import load_pretrained

base, tok = load_pretrained("tiny-gpt")
chat, _ = load_pretrained("tiny-gpt-chat")
ids = torch.tensor([tok.encode("<user> Describe the Zephyr tent . <assistant>")])
print("unknown words:",
      [w for w in tok.tokenize("<user> Describe the Zephyr tent . <assistant>")
       if w not in tok.stoi])
for name, m in [("base", base), ("chat", chat)]:
    torch.manual_seed(0)
    out = m.generate(ids, 25, temperature=0.7,
                     stop_id=tok.token_id("<end>"))[0, ids.shape[1]:].tolist()
    print(f"{name}: {tok.decode(out)!r}")
In the lesson you can change this and run it.

The base model has never seen <user> or <assistant>, so it treats them as noise and drifts into catalog text. The chat model answers the request in form: a complete description, "The Cove tent is made from silnylon...", ending cleanly at <end>. "Zephyr" is not in its vocabulary; it arrived as the unknown token, and the model filled the gap with a real tent's name. The description is fluent, on-brand and about a product nobody asked for.

Quick check
The chat model's training set is built from the catalog. What does this print?
chats = [("Describe the Harbor camp mug .",
          "Meet the Harbor camp mug, made from enamel steel."),
         ("Describe the Ridgeline tent .",
          "Our Ridgeline tent is made from recycled ripstop nylon.")]
declines = [a for q, a in chats if "don't know" in a or "no information" in a]
print(len(chats), len(declines))
  1. 0 2
  2. 2 2
  3. 2 0
  4. 2 1
Show the answer

2 0

Two training chats, neither of which declines. The real set has 300 and the count of declines is the same: zero. A model learns the moves its data contains.

What the data taught

Supervised fine-tuning lowers the loss on the assistant's tokens in each example. Every example's assistant turn begins with the start of a description ("The", "Our", "Meet") and ends with <end> after a full description. So after <assistant>, the model learns that the next token is almost certainly one of those three openers, for any request, because every request in the data was followed by one. The alternative continuation, <end> straight away, which would be the shortest possible refusal, never appeared, so its probability was pushed toward zero.

The next cell reads the chat model's distribution for the first token of its reply, for two real products and two invented ones, and the probability it gives to stopping immediately. Then it asks about twenty invented products and counts how many times the model declined or stopped early.

Try it now

The chat model's first-token shares for the Zephyr tent, reproduced with four logits. <end> sits far below. Raise its logit and watch how far it has to climb before declining becomes the likely move.

In your own words
In your own words: why does supervised fine-tuning on helpful examples make a model unable to decline?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: read the first token and count the declines

Purpose. Read the chat model's first-token chances, then count what a model would learn from two training sets.

Worked example

The first token of every reply

Top three first tokens after <assistant> and the probability of <end>, for known and invented products; then twenty invented products at temperature 0.7.

Uses the setup above, 5 lines.

import re
import requests

text = requests.get("https://api.gradient.test/v1/corpus").json()["text"]
names = sorted({m.group(1)
                for m in (re.match(r"^(?:Meet the|Our|The) ([A-Z][a-z]+)", d)
                          for d in text.strip().split("\n")) if m})

for product in ["Ridgeline tent", "Harbor camp mug", "Zephyr tent", "Falcon headlamp"]:
    with torch.no_grad():
        ids = torch.tensor([tok.encode(f"<user> Describe the {product} . <assistant>")])
        p = torch.softmax(chat(ids)[0, -1], dim=-1)
    top = p.topk(3)
    known = product.split()[0] in names
    firsts = ", ".join(f"{tok.vocab[i]} {v:.2f}"
                       for v, i in zip(top.values.tolist(), top.indices.tolist()))
    print(f"{'known   ' if known else 'invented'} {product:<15} first token: {firsts}; "
          f"p(<end>) {p[tok.token_id('<end>')].item():.4f}")

stopped = opened_with_real_name = 0
for name in ["Zephyr", "Falcon", "Orbit", "Quartz", "Nimbus"]:
    for kind in ["tent", "daypack", "camp mug", "headlamp"]:
        ids = torch.tensor(
            [tok.encode(f"<user> Describe the {name} {kind} . <assistant>")])
        torch.manual_seed(0)
        out = chat.generate(ids, 30, temperature=0.7, stop_id=tok.token_id("<end>"))
        words = [tok.vocab[i] for i in out[0, ids.shape[1]:].tolist()]
        stopped += "<end>" in words[:3]
        opened_with_real_name += any(w in names for w in words[:4])
print(f"20 invented products: stopped within three tokens {stopped}, opened with a "
      f"real catalog name {opened_with_real_name}")
In the lesson you can change this and run it.

For a real product and an invented one alike, the first token is "The", "Meet" or "Our", and the probability of stopping is 0.0000 to four decimals. Across twenty invented products the model never stopped early and opened with a real catalog name every time. It is not lying; it has one behavior, "describe", and no representation of the fact that the request cannot be met.

Quick check
What would have to be true of the fine-tuning data for the chat model to give <end> a real share right after <assistant>?
  1. Some examples would need to stop at once or begin with a decline, giving the loss a reason to raise them
  2. The learning rate would have to be higher so rare tokens get a share
  3. The unknown token would have to appear in the user's turn of some examples
  4. The examples would need longer descriptions, so the stop token appears more often in them
Show the answer

Some examples would need to stop at once or begin with a decline, giving the loss a reason to raise them

A token's probability after a context rises only when training examples show it there. Declines and immediate stops never appear after <assistant> in the data, so nothing lifts them.

From compliance to flattery

Production assistants add a second stage, preference training (Instruction and preference data and RLHF). Pairs of answers are shown to raters, the preferred one is recorded, a reward model learns to predict those preferences, and the policy is trained to score well on it. Raters prefer answers that are complete, confident and agreeable. They rarely prefer "I don't know", and they often prefer the answer that agrees with the view the question implied.

So the pressure that made the tiny chat model describe every product continues at scale and acquires a direction: toward confidence, and toward agreement. Studies of assistants find that they change a correct answer when the user pushes back, mirror the user's stated opinion, and praise work more warmly when the user says it is their own. The alignment lesson calls this sycophancy. It is not a bug in one model; it is what a reward learned from human approval selects for, unless the data and the reward are built to resist it.

Quick check
From the RLHF lesson: what is the reward model trained on?
  1. The number of tokens in each answer, so that longer answers score higher
  2. Human judgments of which of two answers is better, so it predicts what people prefer
  3. Whether the answer declines when uncertain, which raters mark as the safe choice
  4. The correctness of answers, checked against a database of facts kept by the lab
Show the answer

Human judgments of which of two answers is better, so it predicts what people prefer

The reward model learns human preferences between answers. Anything people tend to prefer, including confidence and agreement, becomes part of what it rewards, which is the root of flattery.

Drill

How often does a dataset decline?

Write refusal_rate(answers, phrases): the share of answers that contain any of the given phrases, compared case-insensitively. Return 0.0 for an empty list.

Uses the setup above, 5 lines.

def refusal_rate(answers, phrases):
    return 0.0

answers = ["Our Ridgeline tent is made from recycled ripstop nylon.",
           "I don't know of a product called the Zephyr tent.",
           "Meet the Harbor camp mug, made from enamel steel."]
print(refusal_rate(answers, ["i don't know", "no information"]))
In the lesson, your code is checked as you write it.

Teaching "I don't know"

Since the behavior comes from the data, so does the fix. A model learns to decline when its fine-tuning data contains declines in the right places: requests about things the model cannot know, paired with answers that say so, at a rate that reflects how often such requests arrive. Too few and the model never learns the move; too many and it declines things it does know, which is its own failure. The right rate is a measurement, not a guess, and the hallucination lesson's grounding plus a schema with an explicit unknown value is the version of this you can ship without fine-tuning anything.

The next cell simulates the data side on a tiny scale: two training sets, one with no declines and one where a fifth of the examples decline, and the first-token frequencies a model would learn from each. It is only counting, which is what the loss is doing with far more care.

Code flow. The count behind the pictures is a loop. Step through it on three short answers.

Worked example

What a model would learn from each training set

First-token frequencies of the assistant's turn in two tiny training sets: the catalog's chats as they are, and the same chats with declines added for products that do not exist.

Uses the setup above, 5 lines.

from collections import Counter

describe = ["The Ridgeline tent is made from recycled ripstop nylon.",
            "Meet the Harbor camp mug, made from enamel steel.",
            "Our Cedar hiking pants are made from organic cotton.",
            "The Meadow daypack is made from recycled ripstop nylon."] * 5
declines = ["Sorry, I have no information about that product."] * 5

for name, answers in [("catalog chats only", describe),
                      ("with declines", describe + declines)]:
    first = Counter(a.split()[0].strip(",") for a in answers)
    shares = ", ".join(f"{w} {n / len(answers):.2f}" for w, n in first.most_common())
    print(f"{name} ({len(answers)} examples): {shares}")
It runs right here in your browser, with nothing to install.

With the catalog chats alone, the first token is "The", "Meet" or "Our" in every example, so a model fitted to this data gives "Sorry" no share at all. Add five declines to twenty descriptions and "Sorry" becomes a fifth of the first tokens: the model now has a path to decline, and the fine-tuning loss will build it. Whether it declines the right requests depends on which requests the declines were attached to, which is the data design problem behind every honest assistant.

Quick check
A team wants their fine-tuned support model to say "I don't know" when a question is outside its documents. Which change makes that possible?
  1. Add "be honest" to the system prompt, so the model knows that declining is allowed
  2. Lower the temperature so the model stops guessing when it is unsure
  3. Add training examples of out-of-scope questions paired with declining answers, at a realistic rate
  4. Remove the <end> token so the model cannot stop early and has to think
Show the answer

Add training examples of out-of-scope questions paired with declining answers, at a realistic rate

The move has to exist in the data for the loss to build it. Instructions and sampling settings cannot create a continuation the model was never trained toward.

Drill

Should this request be declined?

Write needs_abstention(request, known_products): return True when the request asks about a product that is not in known_products. A request has the form "Describe the <Name> <type> .", and a known product is a string like "Ridgeline tent". Compare the words between "the" and the final ".".

Uses the setup above, 5 lines.

def needs_abstention(request, known_products):
    return False

known = {"Ridgeline tent", "Harbor camp mug", "Cedar hiking pants"}
print(needs_abstention("Describe the Zephyr tent .", known),
      needs_abstention("Describe the Harbor camp mug .", known))
In the lesson, your code is checked as you write it.
In your own words
Why is "add 20% declines to the training data" not a complete answer? What else has to be right?

Write a sentence or two, then compare it with a good answer in the lesson.

In production: what this justifies

Purpose. Design abstention into the data, the output format and the evaluation.

Never expect a model to volunteer ignorance. Its default, from the data, is to answer. Design the abstention: a schema with an unknown value, grounding so that "the source does not say" is a visible fact in the context, and instructions that make declining the expected completion in that case.

Evaluate abstention separately. An evaluation set that only counts right answers rewards a model that answers everything. Include questions with no answer and score two things: accuracy on the answerable ones and the false-answer rate on the rest. The next cell computes both for a model that always answers.

When fine-tuning, put declines in the data for unanswerable requests at a measured rate, and when judging answers with a model, instruct the judge to penalize confident fabrication rather than reward completeness.

Worked example

An evaluation that counts abstention

Ten questions, four of them unanswerable. A model that answers everything is scored two ways.

Uses the setup above, 5 lines.

items = [("answerable", True), ("answerable", True), ("answerable", False),
         ("answerable", True), ("answerable", True), ("answerable", False),
         ("unanswerable", None), ("unanswerable", None), ("unanswerable", None),
         ("unanswerable", None)]
answered = [True] * 10          # the model always answers

overall = sum(1 for (kind, correct), a in zip(items, answered)
              if kind == "answerable" and correct) / len(items)
accuracy_on_answerable = (sum(1 for (kind, correct) in items
                              if kind == "answerable" and correct)
                          / sum(1 for kind, _ in items if kind == "answerable"))
false_answers = (sum(1 for (kind, _), a in zip(items, answered)
                     if kind == "unanswerable" and a)
                 / sum(1 for kind, _ in items if kind == "unanswerable"))
print(f"naive score {overall:.0%} | accuracy on answerable "
      f"{accuracy_on_answerable:.0%} | false answers on unanswerable "
      f"{false_answers:.0%}")

import matplotlib.pyplot as plt

names = ["naive score", "accuracy on answerable", "false answers"]
plt.bar(names, [overall, accuracy_on_answerable, false_answers])
plt.ylabel("share")
plt.title("One model, three numbers")
plt.show()
In the lesson you can change this and run it.

The naive score, right answers over all questions, says 40%. Split it and the picture changes: the model is right on two thirds of the questions it could answer, and it fabricated an answer for every question that had none. A model that declined the four unanswerable questions would show the same 67% and 0% false answers, and a naive score that cannot tell the two apart is a score that trains teams to ship the fabricating one.

Quick check
A model declines 3 of 4 unanswerable questions and answers all 6 answerable ones, 4 of them correctly. What does this print?
answerable_correct, answerable = 4, 6
unanswerable_answered, unanswerable = 1, 4
print(round(answerable_correct / answerable, 2),
      round(unanswerable_answered / unanswerable, 2))
  1. 0.67 0.25
  2. 0.67 0.75
  3. 0.4 0.25
  4. 0.5 0.25
Show the answer

0.67 0.25

Accuracy on the answerable questions is 4 of 6, 0.67; the false-answer rate is 1 of 4, 0.25. Reporting both is what makes a decline count for something.

Where this shows up in an engineer's week

A support bot invents an answer to a question outside its documents. You can't fix it with a politer prompt. You add an unknown field to the output, put the source in the context so "not in the source" is visible, and add unanswerable questions to the eval so a fix shows up as a falling false-answer rate.

Goes deeper in

In your own words
Your evaluation set has only answerable questions and the model scores 90%. What does that number fail to tell you, and how would you fix the set?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 4 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

The tiny chat model gives <end> a probability of 0.0000 as the first token of its reply, for real and invented products alike. What produced that?

  1. Fine-tuning data in which every assistant turn begins a description and none stops at once
  2. A rule in the code that forbids stopping early, added to keep answers complete
  3. The base model's training on the catalog, where descriptions never end abruptly
  4. The temperature setting, which pushes the stop token's share toward zero

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 6 · about 35 hours of study

Modern LLM engineering

Post-training, fine-tuning, serving and safety 48 lessons 2 free
  1. Base models vs assistants FreeWhat post-training adds to a pretrained model.30 min
  2. Supervised fine-tuningTeach behavior from example conversations.40 min
  3. Instruction and preference dataThe datasets that shape assistant behavior.30 min
  4. RLHFReward models and reinforcement learning from human feedback.40 min
  5. DPO and direct preference methodsPreference training without a separate reward model.40 min
  6. Constitutional AITraining with AI feedback guided by written principles.35 min
  7. Training reasoning modelsReinforcement learning on verifiable problems.40 min
  8. Prompting vs RAG vs fine-tuningPick the cheapest approach that works.30 min
  9. LoRAFine-tune a big model by training small low-rank updates.40 min
  10. Fine-tune an open modelRun a real fine-tune and evaluate the result.45 min
  11. DistillationTrain a small model to imitate a large one.35 min
  12. Tuning an embedding model for retrievalWhen search fails on your own vocabulary: train an adapter with a contrastive loss on question and passage pairs, test on held-out passages, use batch size and hard negatives, and measure what tuning costs elsewhere.45 min
  13. QuantizationStore weights in fewer bits to run models cheaply.35 min
  14. Quantization in practice: zero points, calibration and formatsScale and zero point, per-channel scales, calibration sets, error on a layer's output, quantization-aware training and the file formats you meet at work.35 min
  15. Serving modelsBatching, throughput and latency in production.35 min
  16. One request through a serving stack: latency and costFollow one request through queue, prefill, a prefix cache, decode and the stream, and price a token from a GPU's hourly cost.35 min
  17. Speculative decodingDraft with a small model, verify with a big one.35 min
  18. Running open models locallyDownload, quantize and run a model on your own machine.30 min
  19. Will it fit? Memory for serving and fine-tuningWeights, KV cache, runtime and adapter states added up for a model card and a GPU, with a calculator.30 min
  20. InterpretabilityLooking inside models to see what they compute.40 min
  21. Confident and wrongWhy a model answers with the same assurance whether it is right, wrong or asked about something that does not exist.30 min
  22. Why it can't spell or count lettersThe model sees token IDs, and an embedding learned from usage carries nothing about spelling.25 min
  23. Copying from the promptIn-context learning as a two-head copying circuit: the base tiny GPT cannot copy a repeated pattern, a model trained on repeats grows an induction head and can.30 min
  24. Why temperature changes the answerThe same prompt at 0, 0.7 and 1.5: deterministic loops, plausible variety, broken grammar, measured on a real model.25 min
  25. Why long prompts diluteAttention is a budget shared by every token, so a fact far from the question loses its share; measured, with the cliff beyond the trained window.25 min
  26. Why it never says "I don't know"A chat model answers every request because its fine-tuning data always did; preference training adds flattery; how to build abstention in.25 min
  27. Benchmarks and their limitsWhat leaderboards measure, and what they miss.38 min
  28. Alignment and safetyMaking capable models behave as intended.35 min
  29. Evaluating agentsMeasure multi-step, tool-using behavior.35 min
  30. Reading AI papersGet the idea out of a research paper in 30 minutes.30 min
  31. Capstone: a tuned, evaluated assistantFine-tune, evaluate and deploy a specialist assistant.45 min
  32. Weight-space programming: what it is, and what has been shown Free OptionalA language model reads and writes another network's weights with no training loop: the published experiments, their limits and the criticism, and one edit you make by hand.30 min
  33. Anatomy of a checkpoint OptionalA model file is a set of named tensors: count them, find the ones that write to the residual stream, and save one to bytes and back.35 min
  34. Reading weights without running the model OptionalSingular values, and an attention head's token-to-token circuit, read from the matrices alone and then checked against a measurement.35 min
  35. Writing weights by hand OptionalSet a small network's 55 weights so it catches every ball in a paddle game, the way a program would be written, and meet compiled transformers.40 min
  36. Edits are deltas: task vectors, patches and hashes OptionalSubtract a base from a fine-tune and the edit becomes an object: scale it, take it away, ship it as a patch, and prove by hash what was applied.35 min
  37. Merging models OptionalAverage two fine-tunes, add their task vectors, try SLERP, TIES and DARE, measure the interference, and compare with a model trained on both tasks.40 min
  38. Low-rank edits OptionalCut a fine-tune's change down to its largest directions with the SVD: how few a behavior needs, and how to pull an adapter out of a delta.35 min
  39. Directions: steer the run or edit the weights OptionalFind a behavior's direction in the residual stream, add it at run time, project it out of the weights, and see the effect change from input to input.40 min
  40. Editing one fact OptionalA rank-one change to one matrix rewrites what a product is made from; then reliability, generalization, locality, and what breaks after many edits.40 min
  41. The editor's harness OptionalWhat an LLM that edits weights is given and what it is denied: weights as text and their cost in tokens, a sandbox, a budget of forward passes, and rules the harness enforces.40 min
  42. An LLM reads a network's weights OptionalSix recorded sessions: an LLM is handed 1,315 weights as text and asked what the network was trained to do. Score each answer against the network, and study one it got wrong.40 min
  43. An LLM fixes a model's weights OptionalFind a tiny GPT's loop by hand, then replay recorded sessions in which an LLM hunts for it with ten forward passes, and measure every fix on held-out questions.45 min
  44. An LLM writes a network's weights OptionalFive recorded sessions: an LLM writes every weight of a paddle network from the game's source and one sentence. Score each against its specification, test its design claims, and edit one by hand.40 min
  45. Did it work? Testing an edit OptionalReview a patch for a tiny GPT that can't stop talking: held-out questions, a paired comparison, a same-size control, and what the edit cost.40 min
  46. The published edit, on a real model OptionalRead the edit published for an open model of 870 million weights, apply and verify one in the same form, see what 16-bit and 4-bit storage do to a fix, and run the real thing from a notebook.45 min
  47. Safety, licenses and provenance OptionalBefore a custom model is released: trace its lineage by hash, work out what its parents' licenses pass on, see what an edit can do to a safeguard, and write its card.40 min
  48. Capstone: ship a custom model OptionalSet release gates, then diagnose, edit, measure and document a tiny GPT with a fault no lesson has fixed, and write the record a stranger could check.45 min
Part of a free course 6 lessons

Why Models Do That: six behaviors, traced to the mechanism

Confident wrong answers, miscounted letters, copying from the prompt, temperature, long prompts that dilute, and a model that never says "I don't know": each behavior reproduced on a tiny GPT with every number printed by code, then the mechanism behind it and the Phase 1 technique it justifies.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write