Index Ai Academy
Phase 6 · Why models do that

Why long prompts dilute

Attention is a budget shared by every token, so a fact far from the question loses its share; measured, with the cliff beyond the trained window.

About 25 minutes · Model names checked November 2026 · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers9

Why it matters

A fact at the top of a long prompt gets ignored; the same fact next to the question is used. Models with huge context windows still answer best from the start and the end of the prompt and worst from the middle, and retrieval pipelines that stuff twenty documents into the context do worse than ones that pass three good ones. The long-context lesson described these findings; this lesson shows the mechanism on the course's tiny GPT, with numbers you compute yourself.

The mechanism is arithmetic. Each attention row is a softmax over every token so far, and its weights sum to 1, so every token added to the prompt competes for the same budget; a fact keeps its share only if its score stands out more as the prompt grows. Measure the decay, and prompt design stops being superstition.

What you'll be able to do

  • Measure how a token's share of attention shrinks as more text sits between it and the question
  • Explain the softmax budget: more tokens, smaller shares, unless the scores pull apart
  • Show what happens at positions a model never trained on, and why context limits are real limits
  • Place facts and questions in a prompt, and chunk retrieved text, with the mechanism in mind
Quick check
A product's name sits near the start of a prompt and the question about it comes last. More sentences are inserted between them. A guess at what happens to the attention the question pays to the name:
  1. It shrinks, because every attention row shares a fixed total of 1 across all the tokens so far
  2. It stays the same, since attention finds the name wherever it is and weights it by content
  3. It becomes exactly zero after 10 tokens, the reach of a single attention head
  4. It grows, because the name is now more important by contrast with the filler
Show the answer

It shrinks, because every attention row shares a fixed total of 1 across all the tokens so far

Attention weights are a softmax: they sum to 1 over every token in the context. Each new token takes a share, and unless the name's score pulls further ahead, its share falls. You will measure the fall in a moment.

The idea: one softmax, one budget

Purpose. Why a fact loses attention as a prompt grows, with the arithmetic that shows it.

This lesson rests on Scaled dot-product attention: the weights in one attention row come from a softmax, so they add to 1.

Inside one attention head, the last token computes a score against every earlier token and turns the scores into weights with a softmax (Scaled dot-product attention). Those weights sum to 1. That single fact is the whole mechanism of this lesson: the weights are a budget, and every token in the prompt is a claim on it.

If the scores were all equal, a prompt of 8 tokens would give each one 0.125 and a prompt of 64 would give each 0.016. The scores are not equal, of course; attention is useful because a relevant token can earn a much larger share than the rest. But to keep the same share as the prompt grows, its score has to pull away from the others by more and more. The next cell measures a real share on the tiny GPT as text is inserted between a product's name and the question about it.

Try it now
In your own words
In your own words: why does adding text to a prompt reduce the attention on a fact that has not moved and has not changed?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: measure the budget on the tiny GPT

Purpose. Insert text between a product's name and the question, and read the attention and the prediction.

The setup above loads the model and holds attention_from_last, so the cells show only the measurement. The next cell grows the prompt and records the name's share.

Code flow. The budget is a softmax. Step through it on four scores, one of them 3 higher than the rest.

Worked example

A name's share of attention, as the prompt grows

"Our Ridgeline tent" + k copies of "It is light and roomy." + "is made from". From the last token, the average attention over the four heads of each layer on "Ridgeline", and the share that goes to the last eight tokens.

Setup (runs before every cell) · 29 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import math
import torch
from tinygpt import load_pretrained

model, tok = load_pretrained("tiny-gpt")


def attention_from_last(text):
    """Each layer's attention from the last token, averaged over its heads."""
    ids = torch.tensor([tok.encode(text)])
    rows = []
    with torch.no_grad():
        h = model.tok(ids) + model.pos(torch.arange(ids.shape[1]))
        for blk in model.blocks:
            B, T, d = h.shape
            q, k, _ = blk.qkv(blk.ln1(h)).split(d, dim=-1)
            q, k = (t.view(B, T, blk.h, d // blk.h).transpose(1, 2) for t in (q, k))
            scores = q @ k.transpose(-2, -1) / math.sqrt(d // blk.h)
            mask = torch.tril(torch.ones(T, T)).bool()
            weights = torch.softmax(scores.masked_fill(~mask, float("-inf")), dim=-1)
            rows.append(weights[0, :, -1].mean(dim=0))
            h = blk(h)
    return rows


def p_recycled(text):
    with torch.no_grad():
        p = torch.softmax(model(torch.tensor([tok.encode(text)]))[0, -1], dim=-1)
    return p[tok.token_id("recycled")].item(), p
shares, sizes = [], []
for k in [0, 1, 2, 3, 4, 6]:
    text = "Our Ridgeline tent" + " It is light and roomy." * k + " is made from"
    words = tok.tokenize(text)
    j = words.index("Ridgeline")
    l0, l1 = attention_from_last(text)
    sizes.append(len(words))
    shares.append(float(l0[j]))
    print(f"{len(words):2d} tokens, name {len(words) - 1 - j:2d} back: share on",
          f"Ridgeline layer 0 {l0[j]:.3f}, layer 1 {l1[j]:.3f};",
          f"last eight tokens take {l0[-8:].sum():.2f} of layer 0")

import matplotlib.pyplot as plt

plt.plot(sizes, shares, "o-")
plt.xlabel("tokens in the prompt")
plt.ylabel("share of layer 0's attention on Ridgeline")
plt.title("The name's share shrinks as text is inserted")
plt.show()
In the lesson you can change this and run it.

With nothing in between, "Ridgeline" gets 0.125 of layer 0's attention from the last token, one eighth. One inserted sentence halves that to 0.067; two leave 0.030; six leave 0.008, less than one percent, forty tokens back. Layer 1 drops faster, from 0.027 to nothing. The last column shows where the budget went: with six sentences inserted, the eight most recent tokens hold only a quarter of layer 0's attention, because 42 tokens are now sharing it. Nothing was learned or forgotten between these lines. The prompt got longer.

Quick check
Equal scores, two prompt lengths. What does this print?
import torch

four = torch.softmax(torch.zeros(4), dim=0)
eight = torch.softmax(torch.zeros(8), dim=0)
print(four[0].item(), eight[0].item())
  1. 0.25 0.25
  2. 0.5 0.125
  3. 1.0 1.0
  4. 0.25 0.125
Show the answer

0.25 0.125

With equal scores the budget of 1 is split evenly: a quarter each among four tokens, an eighth among eight. Doubling the prompt halves every share unless the scores change.

Quick check
Equal scores, two prompt lengths. What does this print?
import torch

four = torch.softmax(torch.zeros(4), dim=0)
eight = torch.softmax(torch.zeros(8), dim=0)
print(four[0].item(), eight[0].item())
  1. 0.25 0.125
  2. 1.0 1.0
  3. 0.25 0.25
  4. 0.5 0.125
Show the answer

0.25 0.125

With equal scores the budget of 1 is split evenly: a quarter each among four tokens, an eighth among eight. Doubling the prompt halves every share unless the scores change.

Recency wins on this model

Where did the budget go when the name's share fell? Mostly to the tokens near the end, but less and less to them too: the last eight tokens took all of layer 0's attention in the six-token prompt and a quarter of it in the 42-token one. The tiny GPT was trained on descriptions a few dozen tokens long, where the useful context is almost always close by, so its heads lean toward recent tokens. That is a reasonable habit for its data and a poor one for a long prompt.

The consequence shows in its answers. Insert tent-like sentences between the name and the question and the prediction stays on tent materials. Insert another product's description instead, and the answer follows the nearer product. The next cell puts a camp mug's sentence between "Our Ridgeline tent" and "is made from", once, twice, three times.

Worked example

The nearer product wins

Between the tent's name and the question, k copies of "Meet the Harbor camp mug, made from enamel steel." The probability of the tent's real material, recycled, and the top token.

Uses the setup above, 29 lines.

for k in [0, 1, 2, 3]:
    text = ("Our Ridgeline tent"
            + " Meet the Harbor camp mug, made from enamel steel." * k
            + " is made from")
    with torch.no_grad():
        p = torch.softmax(model(torch.tensor([tok.encode(text)]))[0, -1], dim=-1)
    print(f"k={k}, {len(tok.tokenize(text)):2d} tokens: p(recycled) "
          f"{p[tok.token_id('recycled')].item():.2f}, top token "
          f"{tok.vocab[int(p.argmax())]}")
In the lesson you can change this and run it.

With no mug in the way, recycled has 0.46 (the model's usual near tie between the two tent materials). One mug sentence pushes it down to 0.26; with two the top token becomes "enamel", the mug's material, and recycled is at 0.00. The question is still about the Ridgeline tent, named at the start, but the attention that reaches the start is a few percent and the attention on the recent mug sentences is most of the budget. The model answers about what is near.

Quick check
With two camp-mug sentences between "Our Ridgeline tent" and "is made from", the model's top token was "enamel". Which reading is right?
  1. The nearer product's text holds most of the attention budget, so the model continues its pattern
  2. The model has decided the Ridgeline tent is made of enamel after reading the mug sentences
  3. Repeating a sentence twice makes it true for the model, whatever it is about
  4. The mug sentences overwrote the tent's embedding in the forward pass
Show the answer

The nearer product's text holds most of the attention budget, so the model continues its pattern

The question still names the tent, but the share of attention reaching the name is a few percent; the recent mug sentences dominate the row, and the prediction follows them.

Beyond the window

There is a second limit, and it is not about sharing. A model learns a vector for every position it sees during training (the positional encodings in "How transformers think"). The tiny GPT was trained on windows of 48 tokens, so positions 0 to 47 have trained vectors and positions 48 to 63, although they exist in the table, were never adjusted from their random start. A prompt longer than 48 tokens puts its last words on untrained positions.

The next cell takes the growing prompt from before one step further: seven inserted sentences make 48 tokens, eight make 54. Watch the prediction. Production models have the same cliff at their trained length, softened by better positional schemes but not removed, which is why a context window is a hard limit and why quality degrades toward it even before the limit is reached.

Worked example

Forty-eight tokens, then fifty-four

The tent prompt with seven and then eight inserted sentences. Positions from 48 on were never seen in training.

Uses the setup above, 29 lines.

for k in [6, 7, 8]:
    text = "Our Ridgeline tent" + " It is light and roomy." * k + " is made from"
    ids = tok.encode(text)
    with torch.no_grad():
        p = torch.softmax(model(torch.tensor([ids]))[0, -1], dim=-1)
    top = p.topk(2)
    tops = ", ".join(f"{tok.vocab[i]} {v:.2f}"
                     for v, i in zip(top.values.tolist(), top.indices.tolist()))
    print(f"{len(ids)} tokens: p(recycled) {p[tok.token_id('recycled')].item():.2f}; "
          f"top {tops}")
In the lesson you can change this and run it.

At 42 and 48 tokens the model still predicts a tent material, recycled or silnylon, with the two near a tie as usual. At 54 tokens, with the question sitting on positions it never trained on, the prediction falls apart: recycled at 0.08 and the top token "for", which does not follow "made from" anywhere in the training text. The dilution before this point was gradual; the untrained positions are a cliff.

Plot it. The same prompt for 0 to 8 inserted sentences, with the trained window marked. The chance of the tent's real material falls, then the prediction breaks past 48 tokens.

Uses the setup above, 29 lines.

import matplotlib.pyplot as plt

sizes, chances = [], []
for k in range(0, 9):
    text = "Our Ridgeline tent" + " It is light and roomy." * k + " is made from"
    chance, _ = p_recycled(text)
    sizes.append(len(tok.encode(text)))
    chances.append(chance)
    print(sizes[-1], f"{chance:.2f}")
plt.plot(sizes, chances, "o-")
plt.axvline(47.5, color="gray", linestyle="--", label="end of trained positions")
plt.xlabel("tokens in the prompt")
plt.ylabel("chance of recycled")
plt.legend()
plt.show()
Quick check
From the long-context lesson: when a model is given many retrieved documents and the answer is in one of them, where in the context is the answer found least reliably?
  1. In the middle; the start and the end are recalled best
  2. At the very end
  3. Position makes no difference once the window is large enough
  4. At the very start
Show the answer

In the middle; the start and the end are recalled best

The published finding is a U-shaped curve: recall is high for information at the beginning and the end of the context and lowest in the middle. This lesson's measurements show the end half of that curve, recency, and the budget behind both halves.

Drill

One token's share of the budget

Write share_of(scores, i): the softmax weight of position i given a list of attention scores (one per token). Use NumPy, subtract the maximum score before exponentiating, and return a plain float.

Uses the setup above, 29 lines.

import numpy as np

def share_of(scores, i):
    return 0.0

print(round(share_of([2.0, 0.0, 0.0, 0.0], 0), 3))
In the lesson, your code is checked as you write it.
In your own words
Why is the trained window a hard limit even though the position table has more entries?

Write a sentence or two, then compare it with a good answer in the lesson.

In production: long prompts in large models

Purpose. Place facts and retrieved chunks with the budget in mind, and test long-context features before you trust them.

Lost in the middle, in large models

The tiny GPT leans toward recent tokens because its training text is short. Large models trained on long documents show a two-sided version: they attend strongly to the first tokens as well, partly because early positions are visible to every later token and become a resting place for attention that has nowhere better to go, and partly because documents often state their subject up front. Add recency, and you get the U-shaped curve from the long-context lesson: facts at the start and the end are used, facts in the middle lose the budget contest on both sides.

Position encodings add a second effect. Schemes that work by relative distance make nearby tokens easier to attend to than distant ones, by construction, which is helpful for grammar and harmful for a fact forty paragraphs back. Training on longer and longer documents pushes the curve down but has not flattened it, and every claim of a very large window should be read with the middle in mind.

Quick check
A model is given twelve retrieved paragraphs and a question. The answer is in paragraph six. Which mechanism makes this the hardest placement?
  1. Paragraph six is tokenized differently from the others because of where it sits
  2. Even-numbered paragraphs are weighted less by the positional encoding
  3. It competes with eleven others for the budget and sits far from both the start and the recent end
  4. The model reads only the first and last paragraphs by design, skipping the middle
Show the answer

It competes with eleven others for the budget and sits far from both the start and the recent end

Nothing about the paragraph is special. Its position is: far from the recency that favors the end and from the start that attracts attention on its own, with the whole budget shared twelve ways.

Drill

The budget formula

Write fact_share(n, gap): the attention share of one token whose score exceeds every other token's by gap logits, in a prompt of n tokens. The formula is 1 / (1 + (n − 1)·e^(−gap)). Return a float.

Uses the setup above, 29 lines.

import math

def fact_share(n, gap):
    return 0.0

print(round(fact_share(8, 3), 3), round(fact_share(64, 3), 3))
In the lesson, your code is checked as you write it.

What this justifies

Put what matters where attention reaches it. The key facts first and the question last, with the two as close as the task allows; the advice in Making RAG good to put the best passage nearest the question is this budget in practice.

Retrieve less, better. Twenty documents in the context each take a slice of the budget from the one that holds the answer. Three well-chosen chunks beat twenty, and the chunking rules in Chunking documents exist to make that possible.

Measure recall by position. Before trusting a long-context feature, test it with the needed fact placed at the start, the middle and the end of realistic prompts, and at lengths near the limit. The next cell shows how far a fixed gap carries as the prompt grows, which is the curve such a test will trace.

Worked example

How far a gap carries

A fact whose score beats every other token's by 3 logits, in prompts of 8, 64, 512 and 4,096 tokens.

Uses the setup above, 29 lines.

import math

def fact_share(n, gap):
    return 1 / (1 + (n - 1) * math.exp(-gap))

for n in [8, 64, 512, 4096]:
    print(f"{n:5d} tokens: the fact keeps {fact_share(n, 3):.1%} of the attention")
It runs right here in your browser, with nothing to install.

A gap that is decisive in a short prompt, 74% of the budget among eight tokens, keeps 24% among 64 and 3.8% among 512, and is down to half a percent at 4,096. A model that must still use the fact at 4,096 tokens needs a head whose score gap is about five logits larger there than at eight, and training only partly delivers that. This is the arithmetic behind "put it near the question" and "retrieve less".

Quick check
What does this print?
import math

share = 1 / (1 + 7 * math.exp(-3))
print(round(share, 2), round(share / 2, 2) < 1 / (1 + 63 * math.exp(-3)))
  1. 0.5 True
  2. 0.74 True
  3. 0.74 False
  4. 0.26 False
Show the answer

0.74 False

Among eight tokens the fact keeps 0.74. Going to 64 tokens leaves 0.24, which is less than half of 0.74, so the comparison is False: the share fell by more than half when the prompt grew eightfold.

Where this shows up in an engineer's week

A feature sends a whole manual with every question and answers get worse. Before changing the model, run a position test: put the needed fact at the start, the middle and the end of real prompts at several lengths and record the hit rate in a table. Expect a U shape, then cut the context to the few chunks that matter.

Goes deeper in

In your own words
A team plans to send a 200-page manual with every question. What would you measure first, and what would you expect?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 5 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

On the tiny GPT, the attention on "Ridgeline" from the last token fell from 0.125 to 0.008 as six sentences were inserted before the question. What explains the fall?

  1. Attention is a softmax that sums to 1, so the inserted tokens took shares while the name's score did not pull ahead
  2. The model forgot the name after a few sentences, the way a short-term memory drops old items
  3. The inserted sentences were about tents, which confused the model about the product
  4. The name moved to a position the model had never trained on, past the window

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 6 · about 35 hours of study

Modern LLM engineering

Post-training, fine-tuning, serving and safety 48 lessons 2 free
  1. Base models vs assistants FreeWhat post-training adds to a pretrained model.30 min
  2. Supervised fine-tuningTeach behavior from example conversations.40 min
  3. Instruction and preference dataThe datasets that shape assistant behavior.30 min
  4. RLHFReward models and reinforcement learning from human feedback.40 min
  5. DPO and direct preference methodsPreference training without a separate reward model.40 min
  6. Constitutional AITraining with AI feedback guided by written principles.35 min
  7. Training reasoning modelsReinforcement learning on verifiable problems.40 min
  8. Prompting vs RAG vs fine-tuningPick the cheapest approach that works.30 min
  9. LoRAFine-tune a big model by training small low-rank updates.40 min
  10. Fine-tune an open modelRun a real fine-tune and evaluate the result.45 min
  11. DistillationTrain a small model to imitate a large one.35 min
  12. Tuning an embedding model for retrievalWhen search fails on your own vocabulary: train an adapter with a contrastive loss on question and passage pairs, test on held-out passages, use batch size and hard negatives, and measure what tuning costs elsewhere.45 min
  13. QuantizationStore weights in fewer bits to run models cheaply.35 min
  14. Quantization in practice: zero points, calibration and formatsScale and zero point, per-channel scales, calibration sets, error on a layer's output, quantization-aware training and the file formats you meet at work.35 min
  15. Serving modelsBatching, throughput and latency in production.35 min
  16. One request through a serving stack: latency and costFollow one request through queue, prefill, a prefix cache, decode and the stream, and price a token from a GPU's hourly cost.35 min
  17. Speculative decodingDraft with a small model, verify with a big one.35 min
  18. Running open models locallyDownload, quantize and run a model on your own machine.30 min
  19. Will it fit? Memory for serving and fine-tuningWeights, KV cache, runtime and adapter states added up for a model card and a GPU, with a calculator.30 min
  20. InterpretabilityLooking inside models to see what they compute.40 min
  21. Confident and wrongWhy a model answers with the same assurance whether it is right, wrong or asked about something that does not exist.30 min
  22. Why it can't spell or count lettersThe model sees token IDs, and an embedding learned from usage carries nothing about spelling.25 min
  23. Copying from the promptIn-context learning as a two-head copying circuit: the base tiny GPT cannot copy a repeated pattern, a model trained on repeats grows an induction head and can.30 min
  24. Why temperature changes the answerThe same prompt at 0, 0.7 and 1.5: deterministic loops, plausible variety, broken grammar, measured on a real model.25 min
  25. Why long prompts diluteAttention is a budget shared by every token, so a fact far from the question loses its share; measured, with the cliff beyond the trained window.25 min
  26. Why it never says "I don't know"A chat model answers every request because its fine-tuning data always did; preference training adds flattery; how to build abstention in.25 min
  27. Benchmarks and their limitsWhat leaderboards measure, and what they miss.38 min
  28. Alignment and safetyMaking capable models behave as intended.35 min
  29. Evaluating agentsMeasure multi-step, tool-using behavior.35 min
  30. Reading AI papersGet the idea out of a research paper in 30 minutes.30 min
  31. Capstone: a tuned, evaluated assistantFine-tune, evaluate and deploy a specialist assistant.45 min
  32. Weight-space programming: what it is, and what has been shown Free OptionalA language model reads and writes another network's weights with no training loop: the published experiments, their limits and the criticism, and one edit you make by hand.30 min
  33. Anatomy of a checkpoint OptionalA model file is a set of named tensors: count them, find the ones that write to the residual stream, and save one to bytes and back.35 min
  34. Reading weights without running the model OptionalSingular values, and an attention head's token-to-token circuit, read from the matrices alone and then checked against a measurement.35 min
  35. Writing weights by hand OptionalSet a small network's 55 weights so it catches every ball in a paddle game, the way a program would be written, and meet compiled transformers.40 min
  36. Edits are deltas: task vectors, patches and hashes OptionalSubtract a base from a fine-tune and the edit becomes an object: scale it, take it away, ship it as a patch, and prove by hash what was applied.35 min
  37. Merging models OptionalAverage two fine-tunes, add their task vectors, try SLERP, TIES and DARE, measure the interference, and compare with a model trained on both tasks.40 min
  38. Low-rank edits OptionalCut a fine-tune's change down to its largest directions with the SVD: how few a behavior needs, and how to pull an adapter out of a delta.35 min
  39. Directions: steer the run or edit the weights OptionalFind a behavior's direction in the residual stream, add it at run time, project it out of the weights, and see the effect change from input to input.40 min
  40. Editing one fact OptionalA rank-one change to one matrix rewrites what a product is made from; then reliability, generalization, locality, and what breaks after many edits.40 min
  41. The editor's harness OptionalWhat an LLM that edits weights is given and what it is denied: weights as text and their cost in tokens, a sandbox, a budget of forward passes, and rules the harness enforces.40 min
  42. An LLM reads a network's weights OptionalSix recorded sessions: an LLM is handed 1,315 weights as text and asked what the network was trained to do. Score each answer against the network, and study one it got wrong.40 min
  43. An LLM fixes a model's weights OptionalFind a tiny GPT's loop by hand, then replay recorded sessions in which an LLM hunts for it with ten forward passes, and measure every fix on held-out questions.45 min
  44. An LLM writes a network's weights OptionalFive recorded sessions: an LLM writes every weight of a paddle network from the game's source and one sentence. Score each against its specification, test its design claims, and edit one by hand.40 min
  45. Did it work? Testing an edit OptionalReview a patch for a tiny GPT that can't stop talking: held-out questions, a paired comparison, a same-size control, and what the edit cost.40 min
  46. The published edit, on a real model OptionalRead the edit published for an open model of 870 million weights, apply and verify one in the same form, see what 16-bit and 4-bit storage do to a fix, and run the real thing from a notebook.45 min
  47. Safety, licenses and provenance OptionalBefore a custom model is released: trace its lineage by hash, work out what its parents' licenses pass on, see what an edit can do to a safeguard, and write its card.40 min
  48. Capstone: ship a custom model OptionalSet release gates, then diagnose, edit, measure and document a tiny GPT with a fault no lesson has fixed, and write the record a stranger could check.45 min
Part of a free course 6 lessons

Why Models Do That: six behaviors, traced to the mechanism

Confident wrong answers, miscounted letters, copying from the prompt, temperature, long prompts that dilute, and a model that never says "I don't know": each behavior reproduced on a tiny GPT with every number printed by code, then the mechanism behind it and the Phase 1 technique it justifies.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write