Index Ai Academy
Phase 6 · Adapting models

Tuning an embedding model for retrieval

When search fails on your own vocabulary: train an adapter with a contrastive loss on question and passage pairs, test on held-out passages, use batch size and hard negatives, and measure what tuning costs elsewhere.

About 45 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

Search over your own documents can fail because the embedding model puts your users' questions in the wrong part of the space. Nobody outside your company writes the way your company does. Keyword search, reranking and measurement fix most systems. When they don't, you can teach the model with pairs of questions and the passages that answer them. This lesson does it in the page on a small scale. It measures what the tuning buys, how many pairs it takes and what it breaks.

What you'll be able to do

  • Recognize when retrieval fails because the embedding model doesn't know your vocabulary
  • Train an adapter with a contrastive loss on pairs of questions and passages
  • Test honestly, on questions about passages the training never saw
  • Explain why the wrong passages in a batch do the teaching, and what hard negatives add
  • Measure what tuning costs on other questions, and choose between the remedies
Quick check
A guess before you start. A general-purpose embedding model powers search over a company's internal documents. Staff type questions full of project code names and house abbreviations, and search returns poor results. What is the most likely reason?
  1. The model never learned what those words mean here
  2. The search index has too many documents
  3. The documents are too long to embed
  4. Embedding search can't handle questions
Show the answer

The model never learned what those words mean here

An embedding model places text according to what it saw in training. A code name or a house abbreviation was either absent or meant something else, so the question lands far from the passage that answers it, even though a colleague would connect them at once. The words in the passage are fine. The model's sense of the question is off, and that can be corrected.

The idea: the model doesn't speak your language

This lesson rests on Image-text models in depth: retrieval, negatives and limits, where each item in a batch is scored against every other and its own partner must win. Here the items are questions and passages.

The setting is a made-up company with 2,000 passages of documentation. A general-purpose embedding model has turned each passage into 32 numbers. Every passage has two questions written about it.

  • A general question uses ordinary wording. The model places it close to its passage.
  • A staff question uses the company's jargon. The model places it somewhere else: displaced in the same systematic way for every question, with some noise on top.

Search works as usual. Embed the question, score it against all 2,000 passages, and rank them.

Setup (runs before every cell) · 74 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import numpy as np

SIZE = 32                                        # numbers in each embedding

def unit(v):
    return v / np.linalg.norm(v, axis=-1, keepdims=True)

def company(n_passages=2000):
    """A made-up company's passages, as a general-purpose embedding model sees them, and
    two questions for each. A general question is worded the way the model expects. A
    staff question uses the company's own jargon, which the model places somewhere else:
    the same distortion for every question, plus some noise."""
    noise = np.random.default_rng(0).normal(0, 1, (SIZE, SIZE))
    jargon = 0.3 * np.eye(SIZE) + 0.7 * noise / np.sqrt(SIZE)
    rng = np.random.default_rng(1)
    passages = unit(rng.normal(0, 1, (n_passages, SIZE)))
    staff = unit(passages @ jargon.T + 0.1 * rng.normal(0, 1, (n_passages, SIZE)))
    general = unit(passages + 0.1 * rng.normal(0, 1, (n_passages, SIZE)))
    return passages, staff, general

passages, staff, general = company()
# question i is about passage i
train, held_out = np.arange(0, 1000), np.arange(1000, 2000)

def score(adapter, questions, ids):
    """Search all 2,000 passages for each question.
    Returns the hit rate at 1, the hit rate at 5, and the MRR."""
    scores = (questions[ids] @ adapter.T) @ passages.T
    # 1 means the right passage came first
    rank = (scores > scores[np.arange(len(ids)), ids][:, None]).sum(axis=1) + 1
    return (float((rank == 1).mean()), float((rank <= 5).mean()),
            float((1 / rank).mean()))

# an adapter that leaves every question where it was
no_change = np.eye(SIZE)

def contrastive(adapter, Q, P, temperature=0.1):
    """Each question in the batch should score highest with its own passage, and lower
    with everyone else's."""
    # row i: question i against every passage in the batch
    S = (Q @ adapter.T) @ P.T / temperature
    S = S - S.max(axis=1, keepdims=True)
    # softmax along each row
    chance = np.exp(S) / np.exp(S).sum(axis=1, keepdims=True)
    own = np.arange(len(Q))
    loss = float(-np.log(chance[own, own] + 1e-12).mean())
    # the gradient of the loss with respect to the scores
    chance[own, own] -= 1
    return loss, (chance @ P / temperature / len(Q)).T @ Q

def tune(pairs, steps=300, batch=32, rate=0.2, pull=0.0, mix=None, hard=0, report=()):
    """Learn a matrix that is applied to questions only. The passages, and the search
    index built from them, stay as they are."""
    rng, adapter = np.random.default_rng(0), np.eye(SIZE)
    for step in range(1, steps + 1):
        rows = rng.choice(pairs, min(batch, len(pairs)), replace=False)
        Q, P = staff[rows], passages[rows]
        # add as many general questions, with their passages
        if mix is not None:
            more = rng.choice(mix, min(batch, len(mix)), replace=False)
            Q, P = np.vstack([Q, general[more]]), np.vstack([P, passages[more]])
        # add the wrong passages that currently score highest
        if hard:
            S = (Q @ adapter.T) @ passages[pairs].T
            S[np.arange(len(rows)), np.searchsorted(pairs, rows)] = -1e9
            P = np.vstack([P,
                           passages[pairs[np.argsort(-S, axis=1)[:, :hard].ravel()]]])
        loss, gradient = contrastive(adapter, Q, P)
        adapter = adapter - rate * (gradient + pull * (adapter - np.eye(SIZE)))
        if step in report:
            print("   step", str(step).rjust(3), "| loss", f"{loss:.3f}",
                  "| hit rate at 1 on held-out staff questions",
                  f"{score(adapter, staff, held_out)[0]:.3f}")
    return adapter
print("2,000 passages, and questions about the 1,000 held out for testing")
print("                   | hit rate at 1 | hit rate at 5 | MRR")
for name, questions in (("general questions", general), ("staff questions", staff)):
    r1, r5, mrr = score(no_change, questions, held_out)
    print(name.ljust(18), "|", f"{r1:13.3f}", "|", f"{r5:13.3f}", "|", f"{mrr:.3f}")

The two measures are the ones from the lesson on making retrieval good. The hit rate at k is the share of questions whose right passage is among the top k results. You will also see it called recall at k. The MRR averages 1 divided by the rank of the right passage.

For general questions, search is perfect: the right passage is first every time.

For staff questions, the right passage comes first 3.7% of the time and is in the top five 11.7% of the time. With 2,000 passages a random guess would manage one time in 2,000, so the model is not lost. It is badly off, for exactly the questions this system was built to answer.

Keyword search and reranking, from Making RAG good, are the first things to try, and they are cheaper than what follows. This lesson is about the case where they aren't enough: the right passage is not even among the candidates, so there is nothing for a reranker to promote.

Quick check
From the lesson on making retrieval good: you test search on 10 questions. The right document is ranked 1st for four of them, 2nd for four, and 4th for two. What is the hit rate at 3?
  1. 0.4
  2. 0.2
  3. 1.0
  4. 0.8
Show the answer

0.8

The hit rate at 3 counts the questions whose right document is anywhere in the top three: the four ranked 1st and the four ranked 2nd, which is 8 of 10. The two ranked 4th miss. The MRR for the same results would be (4 × 1 + 4 × 0.5 + 2 × 0.25) / 10 = 0.65, since it also rewards being nearer the top.

Teaching with pairs

The training data is simple to describe: pairs of a real question and the passage that answers it. Support tickets with their resolutions, search logs with the result someone clicked, or questions written by a language model for each passage and checked by a person.

The loss is contrastive. Take a batch of pairs, and score every question against every passage in the batch. Each question's own passage should score highest, and all the others in the batch serve as wrong answers, for free. A softmax along each row turns that into the familiar cross-entropy loss.

What gets trained here is small on purpose: one 32 by 32 matrix, an adapter, applied to questions only. The passages are not touched.

Quick check
From contrastive learning: a batch holds 4 matching pairs. How many wrong answers does each question get from the batch?
  1. 0: the batch holds only right answers
  2. 1: only the closest one
  3. 3: the other three passages
  4. 4: every passage
Show the answer

3: the other three passages

Each question's own passage is the right one. The other 3 passages in the batch are wrong for it, and they come free, with no extra labeling.

Try it now

Read the loss off the table yourself. The cell turns each row of scores into chances with a softmax, then takes minus the log of the chance on the diagonal.

Uses the setup above, 74 lines.

import numpy as np

def batch_loss(scores):
    scores = np.array(scores, dtype=float)
    chance = np.exp(scores) / np.exp(scores).sum(axis=1, keepdims=True)
    return float(-np.log(np.diag(chance)).mean())

before = [[0.5, 0.9, 0.4], [0.8, 0.6, 0.9], [0.4, 0.9, 0.5]]
after = [[2, 1, 0], [1, 2, 1], [0, 1, 2]]
print(round(batch_loss(before), 3), round(batch_loss(after), 3), round(np.log(3), 3))
It runs right here in your browser, with nothing to install.

The last number is the loss of a pure guess among 3 passages, log 3. The untuned batch does worse than that.

Quick check
A batch has two pairs. The scores, question by passage, are below, and each question's own passage is on the diagonal. What does this print: the contrastive loss?
import math

scores = [[2.0, 0.0],
          [0.0, 2.0]]
loss = 0.0
for i, row in enumerate(scores):
    loss += -math.log(math.exp(row[i]) / sum(math.exp(s) for s in row))
print(round(loss / len(scores), 3))
  1. 2.0
  2. 0.127
  3. 0.693
  4. 0.0
Show the answer

0.127

For each question, the chance given to its own passage is e² / (e² + e⁰), about 0.881, and the loss is minus its log, about 0.127. Both rows are alike, so the average is 0.127. The loss would be 0 only if the wrong passage's score were infinitely far below, and 0.693 if the two scores were equal and the model were guessing.

Drill

The contrastive loss for a batch

Write contrastive_loss(scores). scores[i][j] is the score of question i against passage j, and passage i is the right one for question i. For each row, take the softmax and then minus the log of the chance on the diagonal. Return the mean over the rows, or 0.0 for an empty batch. Subtract each row's largest score before using math.exp, so that large scores don't overflow.

Uses the setup above, 74 lines.

import math


def contrastive_loss(scores):
    return 0.0


print(round(contrastive_loss([[2.0, 0.0], [0.0, 2.0]]), 3))
In the lesson, your code is checked as you write it.
In your own words
Why can't a reranker fix a hit rate at 5 of 11.7%, if it only reorders the top five candidates?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: train the adapter and test it honestly

Purpose. Train one 32 by 32 matrix with the loss above, then test it on passages it never saw.

The setup above holds the company, the score function, the loss with its gradient and tune, so each cell shows only the idea.

Uses the setup above, 74 lines.

print("tuning on 1,000 pairs of (staff question, its passage):")
adapter = tune(train, report=(1, 10, 50, 100, 300))
r1, r5, mrr = score(adapter, staff, held_out)
print()
print("staff questions about passages never used in training: hit rate at 1",
      f"{r1:.3f}", "| at 5", f"{r5:.3f}", "| MRR", f"{mrr:.3f}")
print("numbers learned:", adapter.size, "| passages re-embedded:", 0)

The loss falls from 2.253 to 0.027, and the hit rate at 1 on held-out staff questions climbs with it: 0.046, 0.195, 0.556, 0.681, and 0.774 after 300 steps. The right passage is in the top five 93.5% of the time, where it was 11.7%.

Those held-out questions are about passages that never appeared in training. The adapter did not memorize 1,000 answers. It learned how staff wording is displaced, and that carries over.

The last line is why this design is popular. It learned 1,024 numbers, and no passage had to be embedded again. Fine-tuning the embedding model itself can do better still, but every vector in the index must then be recomputed, and the old and new vectors can't be mixed. An adapter on the question side changes nothing that is stored.

Code flow. Step through one update on two pairs in two dimensions. S is the score table, chance its row softmax, and chance[own, own] -= 1 turns each row into the slope of the loss: the own passage is pushed up, the others down. The last two lines move adapter against the gradient.

How many pairs, and how to test honestly

Pairs cost effort to collect, so the first question anyone asks is how many are needed. The cell trains with 10 to 1,000 pairs and measures each adapter twice: on the questions it was trained with, and on questions about held-out passages.

Uses the setup above, 74 lines.

print("hit rate at 1")
print("pairs used | on the training pairs' own questions | on questions about held-out "
      "passages")
for n in (10, 30, 100, 300, 1000):
    adapter = tune(train[:n])
    print(str(n).rjust(10), "|", f"{score(adapter, staff, train[:n])[0]:36.3f}", "|",
          f"{score(adapter, staff, held_out)[0]:.3f}")

Read the right-hand column first, since it is the one that counts. With 10 pairs the hit rate at 1 on held-out passages is 0.088, hardly better than the untouched model's 0.037. With 100 pairs it is 0.484, with 300 it is 0.689, and with 1,000 it is 0.774. Each tripling of the data still helps, by less.

Now the left-hand column. With 10 pairs the adapter scores a perfect 1.000 on its own training questions. A matrix of 1,024 numbers can memorize ten pairs outright. Anyone who tested on those questions would report a solved problem, and ship a system that is right 9% of the time.

With 1,000 pairs the two columns nearly agree, 0.836 and 0.774. That is a good sign: when a model can no longer memorize its training set, what it has learned is the general rule.

The rule for testing follows. Split by passage, not by question. If one passage has five questions about it, and some go into training while others go into the test, the test measures memory of that passage. Hold out whole passages, with every question about them, and the test measures what the tuned model will do for a question nobody has asked yet.

Plot it. The same sweep as a picture: memorizing and learning split apart on a log axis.

Uses the setup above, 74 lines.

import matplotlib.pyplot as plt

sizes = [10, 30, 100, 300, 1000]
adapters = [tune(train[:n]) for n in sizes]
seen = [score(a, staff, train[:n])[0] for a, n in zip(adapters, sizes)]
held = [score(a, staff, held_out)[0] for a in adapters]
plt.plot(sizes, seen, "o--", label="questions it trained on")
plt.plot(sizes, held, "o-", label="questions about held-out passages")
plt.xscale("log")
plt.xlabel("training pairs")
plt.ylabel("hit rate at 1")
plt.title("Memorizing is not learning")
plt.legend()
plt.show()

The dashed line sits at 1.0 for the smallest sets. The solid line is the one that predicts what users will see.

Quick check
An adapter trained on 10 pairs scores a hit rate at 1 of 1.000 on its training questions and 0.088 on questions about held-out passages. Which number describes how the system will perform?
  1. Neither can be known until it is deployed
  2. About 0.54: the average of the two
  3. 0.088: real questions will mostly be about other passages
  4. 1.000: it was measured on real question and passage pairs
Show the answer

0.088: real questions will mostly be about other passages

The training questions are ones the adapter has adjusted itself to fit, and ten pairs are few enough to memorize. A real system faces questions it has never seen, about passages that mostly weren't in the training pairs, which is what the held-out number measures. Averaging in a memorized score only blurs it.

The wrong answers do the teaching

In the contrastive loss, a question learns from the passages it is told not to match, called negatives. In the training above they were simply the other passages in the batch. The cell changes how many there are, and then how hard they are.

Uses the setup above, 74 lines.

print("each question is compared with its own passage and the other passages in its "
      "batch")
print("batch size | wrong passages seen with each question | hit rate at 1, held-out | "
      "at 5")
for batch in (2, 8, 32, 128):
    r1, r5, mrr = score(tune(train, batch=batch), staff, held_out)
    print(str(batch).rjust(10), "|", str(batch - 1).rjust(37), "|", f"{r1:23.3f}", "|",
          f"{r5:.3f}")

r1, r5, mrr = score(tune(train, batch=32, hard=1), staff, held_out)
print()
print("batch of 32, plus for each question the wrong passage that currently scores "
      "highest:")
print("   hit rate at 1, held-out", f"{r1:.3f}", "| at 5", f"{r5:.3f}")

With a batch of 2, each question is compared with one wrong passage, and the hit rate at 1 reaches only 0.294. With 7 wrong passages it is 0.667, with 31 it is 0.774, and with 127 it is 0.821. Nothing else changed: the same pairs and the same number of steps.

A random wrong passage is usually easy to rule out, and an easy comparison teaches little. In a bigger batch, more of the wrong passages happen to be close, and those are the ones that sharpen the model. This is why contrastive training is run with the largest batch the hardware allows.

The last lines go looking for difficulty on purpose. For each question, the wrong passage that currently scores highest is added to the batch: a hard negative. With a batch of 32 that lifts the hit rate at 1 from 0.774 to 0.837, better than a batch of 128 managed with random ones.

Hard negatives carry one risk. In real data the passage that scores highest and isn't the labeled answer is sometimes also a correct answer: a near-duplicate, or a second page covering the same thing. Training the model to push it away teaches it something false. Such false negatives are the usual reason hard-negative mining disappoints, and the reason mined negatives are often filtered or sampled from a little way down the ranking.

Quick check
Contrastive training with a batch of 128 beat a batch of 8, using the same pairs and the same number of steps. Why?
  1. Each question was given more correct passages
  2. A bigger batch uses a larger adapter
  3. The larger batch saw every pair more often per step
  4. Each question was compared against more wrong passages
Show the answer

Each question was compared against more wrong passages

With in-batch negatives, the other passages in the batch are the wrong answers. A batch of 128 sets each question against 127 of them, and some of those will be close calls, which is where the learning happens. There is still exactly one correct passage for each question, and the adapter is the same size.

Drill

Mine hard negatives

Write hard_negatives(scores, targets, k). scores[i][j] is the score of question i against passage j, and targets[i] is the position of question i's right passage. For each question return the positions of the k highest-scoring passages other than its target, highest first. On equal scores, the lower position comes first. If fewer than k are available, return them all.

Uses the setup above, 74 lines.

def hard_negatives(scores, targets, k):
    return [[] for row in scores]


print(hard_negatives([[0.9, 0.8, 0.1, 0.7], [0.2, 0.3, 0.9, 0.4]], [0, 2], 2))
In the lesson, your code is checked as you write it.
In your own words
Why is a hard negative more useful than a random one, and what makes it dangerous?

Write a sentence or two, then compare it with a good answer in the lesson.

In production: what tuning costs, and which remedy to use

So far every number has been about staff questions. The system also gets general questions, which the untouched model answered perfectly. The cell checks on them.

Uses the setup above, 74 lines.

print("hit rate at 1 on held-out questions       | staff | general")
runs = (("before tuning", no_change),
        ("tuned on staff questions", tune(train)),
        ("the same, pulled gently toward no change", tune(train, pull=0.02)),
        ("the same, pulled hard toward no change", tune(train, pull=0.1)),
        ("tuned on staff and general questions", tune(train, mix=train)))
for name, adapter in runs:
    print(name.ljust(41), "|", f"{score(adapter, staff, held_out)[0]:.3f}", "|",
          f"{score(adapter, general, held_out)[0]:.3f}")

The second row is the one to remember. The adapter that took staff questions from 0.037 to 0.774 has taken general questions from 1.000 to 0.474. It was trained only on jargon, so it moves every question as if it were written in jargon. A model tuned for one kind of input and damaged on the rest has undergone what is called catastrophic forgetting, and nothing warned of it, since the training loss looked excellent.

The other rows are two standard remedies.

  • Stay close to the original. Add a penalty that pulls the adapter toward doing nothing. A gentle pull gives 0.676 on staff questions and 0.823 on general ones. A hard pull protects general questions almost completely, 0.993, and gives back most of the gain: 0.423.
  • Train on both. Mix general questions into the training batches. That gives 0.646 and 0.876.

Neither is free. One matrix is being asked to do two jobs, and it settles between them.

Two habits follow from this table. Keep a second test set of the questions that already worked, and report both numbers after every change. And decide what the mix of real traffic is before choosing, which is what the worked example does.

Plot it. What each remedy buys on staff questions and what it costs on general ones.

Uses the setup above, 74 lines.

import matplotlib.pyplot as plt

runs = {"no tuning": no_change, "tuned": tune(train),
        "pulled 0.02": tune(train, pull=0.02), "both kinds": tune(train, mix=train)}
names = list(runs)
on_staff = [score(a, staff, held_out)[0] for a in runs.values()]
on_general = [score(a, general, held_out)[0] for a in runs.values()]
xs = list(range(len(names)))
plt.bar([x - 0.2 for x in xs], on_staff, 0.4, label="staff questions")
plt.bar([x + 0.2 for x in xs], on_general, 0.4, label="general questions")
plt.xticks(xs, names)
plt.ylabel("hit rate at 1, held-out")
plt.title("What tuning buys and what it costs")
plt.legend()
plt.show()
Quick check
After tuning on domain pairs, a retrieval model's hit rate on domain questions rose from 0.04 to 0.77. What else must be measured before it is deployed?
  1. The number of weights that changed
  2. Its hit rate on the training questions
  3. Its training loss on the domain pairs
  4. Its hit rate on other kinds of question
Show the answer

Its hit rate on other kinds of question

Tuning moves the model toward the data it was tuned on, and can move it away from everything else. In this lesson, general questions fell from 1.000 to 0.474 while the training loss looked excellent. Only a separate test set of previously working questions reveals that, so it should be run after every change.

Worked example

Worked example: choosing for real traffic

Which option is best depends on who is asking. Suppose 7 questions in 10 come from staff, in the company's wording, and the rest are general. The cell scores five options on both kinds, and weights them by that traffic. The last option assumes the system can tell the two kinds of question apart, for instance by which page they were asked from.

Uses the setup above, 74 lines.

# 7 questions in 10 come from staff, in the company's own words
staff_share = 0.7

tuned, gentle, both = tune(train), tune(train, pull=0.02), tune(train, mix=train)
def recall(adapter, questions):
    return score(adapter, questions, held_out)[0]

options = (("leave the model alone", recall(no_change, staff),
            recall(no_change, general)),
           ("tuned adapter for everything", recall(tuned, staff),
            recall(tuned, general)),
           ("gently pulled adapter for everything", recall(gentle, staff),
            recall(gentle, general)),
           ("adapter trained on both kinds", recall(both, staff),
            recall(both, general)),
           ("tuned adapter for staff, no adapter for the rest", recall(tuned, staff),
            recall(no_change, general)))
print(
    "option                                           | staff | general | all traffic")
for name, on_staff, on_general in options:
    print(name.ljust(48), "|", f"{on_staff:.3f}", "|", f"{on_general:7.3f}", "|",
          f"{staff_share * on_staff + (1 - staff_share) * on_general:.3f}")
In the lesson you can change this and run it.

Leaving the model alone scores 0.326 across all traffic: perfect on the 30% that is general, and nearly useless on the 70% that isn't. The tuned adapter used for everything reaches 0.684, and the two compromise adapters do a little better, 0.720 and 0.715, by giving up some of the staff gain to protect general questions. The last row beats them all at 0.842, and it involves no further training. It uses the tuned adapter only for staff questions and passes general questions through unchanged, so each kind gets the treatment that suits it: 0.774 and 1.000. This is possible only when the adapter sits on the question side and the stored passages were never altered, so both kinds of question search the same index. Three lessons to take away. Weight results by real traffic before comparing options. A routing rule is often worth more than a cleverer model. And none of these numbers is 1.0: a tuned first stage still benefits from keyword search beside it and a reranker after it.

Quick check
70% of questions are from staff and 30% are general. An adapter has a hit rate at 1 of 0.774 on staff questions and 0.474 on general ones. Routing uses it for staff questions only, and leaves general ones at 1.0. What does this print: the hit rate across all traffic, without and with routing?
print(round(0.7 * 0.774 + 0.3 * 0.474, 3), round(0.7 * 0.774 + 0.3 * 1.0, 3))
  1. 0.624 0.887
  2. 0.774 1.0
  3. 0.684 0.842
  4. 0.684 0.774
Show the answer

0.684 0.842

Each kind of question counts in proportion to its share of traffic. Without routing: 0.7 × 0.774 + 0.3 × 0.474 = 0.684. With routing the general questions keep their original 1.0: 0.7 × 0.774 + 0.3 × 1.0 = 0.842. The staff questions fare the same either way. All of the gain comes from no longer damaging the rest.

Where this shows up in an engineer's week

Retrieval tuning lands on your desk as a bad search report: staff questions return the wrong article. The routine is the same each time. Collect pairs from tickets or search logs, hold out whole articles, tune, and re-measure on staff and general questions before changing the index. The common failure is a test set that shares passages with the training pairs, which makes a memorized adapter look solved. Check that split first.

Goes deeper in

  • LoRA: the same idea, a small trained matrix beside frozen weights, applied to a language model.
  • Making RAG good: keyword search and reranking, the cheaper fixes to try before tuning.
In your own words
A team wants to fine-tune their embedding model "to improve search". What would you ask them before they start?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 9 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 13 review cards for spaced repetition

Try a quiz question

In contrastive training with in-batch negatives, what plays the role of the wrong answers for a question?

  1. The passages paired with the other questions in the batch
  2. Passages written by hand to be misleading
  3. Random strings of text generated for each step
  4. The question itself, embedded a second time

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 6 · about 35 hours of study

Modern LLM engineering

Post-training, fine-tuning, serving and safety 48 lessons 2 free
  1. Base models vs assistants FreeWhat post-training adds to a pretrained model.30 min
  2. Supervised fine-tuningTeach behavior from example conversations.40 min
  3. Instruction and preference dataThe datasets that shape assistant behavior.30 min
  4. RLHFReward models and reinforcement learning from human feedback.40 min
  5. DPO and direct preference methodsPreference training without a separate reward model.40 min
  6. Constitutional AITraining with AI feedback guided by written principles.35 min
  7. Training reasoning modelsReinforcement learning on verifiable problems.40 min
  8. Prompting vs RAG vs fine-tuningPick the cheapest approach that works.30 min
  9. LoRAFine-tune a big model by training small low-rank updates.40 min
  10. Fine-tune an open modelRun a real fine-tune and evaluate the result.45 min
  11. DistillationTrain a small model to imitate a large one.35 min
  12. Tuning an embedding model for retrievalWhen search fails on your own vocabulary: train an adapter with a contrastive loss on question and passage pairs, test on held-out passages, use batch size and hard negatives, and measure what tuning costs elsewhere.45 min
  13. QuantizationStore weights in fewer bits to run models cheaply.35 min
  14. Quantization in practice: zero points, calibration and formatsScale and zero point, per-channel scales, calibration sets, error on a layer's output, quantization-aware training and the file formats you meet at work.35 min
  15. Serving modelsBatching, throughput and latency in production.35 min
  16. One request through a serving stack: latency and costFollow one request through queue, prefill, a prefix cache, decode and the stream, and price a token from a GPU's hourly cost.35 min
  17. Speculative decodingDraft with a small model, verify with a big one.35 min
  18. Running open models locallyDownload, quantize and run a model on your own machine.30 min
  19. Will it fit? Memory for serving and fine-tuningWeights, KV cache, runtime and adapter states added up for a model card and a GPU, with a calculator.30 min
  20. InterpretabilityLooking inside models to see what they compute.40 min
  21. Confident and wrongWhy a model answers with the same assurance whether it is right, wrong or asked about something that does not exist.30 min
  22. Why it can't spell or count lettersThe model sees token IDs, and an embedding learned from usage carries nothing about spelling.25 min
  23. Copying from the promptIn-context learning as a two-head copying circuit: the base tiny GPT cannot copy a repeated pattern, a model trained on repeats grows an induction head and can.30 min
  24. Why temperature changes the answerThe same prompt at 0, 0.7 and 1.5: deterministic loops, plausible variety, broken grammar, measured on a real model.25 min
  25. Why long prompts diluteAttention is a budget shared by every token, so a fact far from the question loses its share; measured, with the cliff beyond the trained window.25 min
  26. Why it never says "I don't know"A chat model answers every request because its fine-tuning data always did; preference training adds flattery; how to build abstention in.25 min
  27. Benchmarks and their limitsWhat leaderboards measure, and what they miss.38 min
  28. Alignment and safetyMaking capable models behave as intended.35 min
  29. Evaluating agentsMeasure multi-step, tool-using behavior.35 min
  30. Reading AI papersGet the idea out of a research paper in 30 minutes.30 min
  31. Capstone: a tuned, evaluated assistantFine-tune, evaluate and deploy a specialist assistant.45 min
  32. Weight-space programming: what it is, and what has been shown Free OptionalA language model reads and writes another network's weights with no training loop: the published experiments, their limits and the criticism, and one edit you make by hand.30 min
  33. Anatomy of a checkpoint OptionalA model file is a set of named tensors: count them, find the ones that write to the residual stream, and save one to bytes and back.35 min
  34. Reading weights without running the model OptionalSingular values, and an attention head's token-to-token circuit, read from the matrices alone and then checked against a measurement.35 min
  35. Writing weights by hand OptionalSet a small network's 55 weights so it catches every ball in a paddle game, the way a program would be written, and meet compiled transformers.40 min
  36. Edits are deltas: task vectors, patches and hashes OptionalSubtract a base from a fine-tune and the edit becomes an object: scale it, take it away, ship it as a patch, and prove by hash what was applied.35 min
  37. Merging models OptionalAverage two fine-tunes, add their task vectors, try SLERP, TIES and DARE, measure the interference, and compare with a model trained on both tasks.40 min
  38. Low-rank edits OptionalCut a fine-tune's change down to its largest directions with the SVD: how few a behavior needs, and how to pull an adapter out of a delta.35 min
  39. Directions: steer the run or edit the weights OptionalFind a behavior's direction in the residual stream, add it at run time, project it out of the weights, and see the effect change from input to input.40 min
  40. Editing one fact OptionalA rank-one change to one matrix rewrites what a product is made from; then reliability, generalization, locality, and what breaks after many edits.40 min
  41. The editor's harness OptionalWhat an LLM that edits weights is given and what it is denied: weights as text and their cost in tokens, a sandbox, a budget of forward passes, and rules the harness enforces.40 min
  42. An LLM reads a network's weights OptionalSix recorded sessions: an LLM is handed 1,315 weights as text and asked what the network was trained to do. Score each answer against the network, and study one it got wrong.40 min
  43. An LLM fixes a model's weights OptionalFind a tiny GPT's loop by hand, then replay recorded sessions in which an LLM hunts for it with ten forward passes, and measure every fix on held-out questions.45 min
  44. An LLM writes a network's weights OptionalFive recorded sessions: an LLM writes every weight of a paddle network from the game's source and one sentence. Score each against its specification, test its design claims, and edit one by hand.40 min
  45. Did it work? Testing an edit OptionalReview a patch for a tiny GPT that can't stop talking: held-out questions, a paired comparison, a same-size control, and what the edit cost.40 min
  46. The published edit, on a real model OptionalRead the edit published for an open model of 870 million weights, apply and verify one in the same form, see what 16-bit and 4-bit storage do to a fix, and run the real thing from a notebook.45 min
  47. Safety, licenses and provenance OptionalBefore a custom model is released: trace its lineage by hash, work out what its parents' licenses pass on, see what an edit can do to a safeguard, and write its card.40 min
  48. Capstone: ship a custom model OptionalSet release gates, then diagnose, edit, measure and document a tiny GPT with a fault no lesson has fixed, and write the record a stranger could check.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write