Index Ai Academy
Phase 6 · Inference

One request through a serving stack: latency and cost

Follow one request through queue, prefill, a prefix cache, decode and the stream, and price a token from a GPU's hourly cost.

About 35 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers8

Why it matters

Serving models showed how batching raises throughput. A user does not feel throughput. They feel the wait before the first word, and the speed of the words after it. A finance person does not see tokens per second either. They see dollars.

This lesson follows one request through a server, stage by stage, with time and money at each step. You will work it out by hand for a 1,200-token prompt and a 200-token reply, then run a simulation that adds a prefix cache, a second replica and a router, and read the numbers a team watches when the load rises.

What you'll be able to do

  • Follow one request through queue, prefill, decode and the stream, and work out its time to first token and its total time
  • Price a request in dollars from a GPU's hourly cost, and say why an idle GPU makes tokens dearer
  • Show with a simulation how a prefix cache, a second replica and a small-then-large router change p50 and p95 latency and cost
  • Read a serving command, a streaming timing call and a metrics line, and plan capacity from an arrival rate
Quick check
A user sends a long prompt to a chat app and waits. A guess: where does most of the wait before the first word come from?
  1. Converting the reply to text
  2. Decoding the first word, which is the slowest step
  3. Downloading the model weights for each request
  4. The queue, plus the prefill pass that reads the whole prompt
Show the answer

The queue, plus the prefill pass that reads the whole prompt

Before the first word the server may wait in a queue, then it reads the whole prompt in one pass called prefill. Decoding each word after that is much quicker per word.

The idea: five stages, and a clock and a meter on each

This lesson rests on Serving models: a decode step takes 10 ms plus 0.4 ms for every conversation in the batch, and continuous batching keeps the seats full.

A request is not one operation. It passes five stages:

  1. Tokenize. The text becomes token numbers. This is fast, and this lesson ignores it.
  2. Queue. The request waits for a seat in the batch.
  3. Prefill. One pass reads the whole prompt and builds its KV cache. A cached start of the prompt skips this work.
  4. Decode. One token per step, until the reply ends.
  5. Stream. Each token goes to the user as it is made.

The user feels two times. Time to first token (TTFT) is the queue plus the prefill. After that, the words arrive one every step.

Assumptions for this lesson. Prefill reads 8,000 prompt tokens a second, and a GPU costs $2.00 an hour. Both are round numbers chosen to keep the sums readable. They are not measurements or a price quote: use your own. The step time is the rule from Serving models.

Quick check
From the lesson on serving: a step takes 10 ms plus 0.4 ms for each conversation in the batch. What does this print for a batch of 10, in milliseconds?
batch = 10
print(10 + 0.4 * batch)
  1. 10.4
  2. 140.0
  3. 14.0
  4. 4.0
Show the answer

14.0

10 ms + 0.4 ms × 10 = 14 ms for each step, whatever the length of each reply.

The three sums

Three formulas cover the whole path:

TTFT=queue+prompt tokens not cachedprefill rate,decode=(reply tokens−1)×step time\text{TTFT} = \text{queue} + \frac{\text{prompt tokens not cached}}{\text{prefill rate}}, \qquad \text{decode} = (\text{reply tokens} - 1) \times \text{step time}
dollars per million tokens=price per hourtokens per hour / 106\text{dollars per million tokens} = \frac{\text{price per hour}}{\text{tokens per hour} \, / \, 10^6}

In plain words: the first token comes out of the prefill, so every later token costs one step. The price of a token is what the GPU costs per hour, divided by how many tokens it makes in that hour.

Take a 1,200-token prompt, a 200-token reply, a batch of 10 and a queue of 120 ms. Prefill takes 1,200 / 8,000 = 0.15 s, which is 150 ms. So TTFT is 120 + 150 = 270 ms. A step takes 10 + 0.4 × 10 = 14 ms. The other 199 tokens take 199 × 14 = 2,786 ms. The whole reply takes 270 + 2,786 = 3,056 ms.

Try it now

The first token is out at 270 ms. The prefill's cost is 150 ms of a whole GPU. The decode's cost is shared by 10 requests.

A prefix cache skips the prefill

Many requests start with the same text: a system prompt, a policy, a long document. A prefix cache keeps the KV cache for that start. If 1,000 of the 1,200 prompt tokens are cached, prefill reads only 200 tokens: 200 / 8,000 = 25 ms, not 150. TTFT falls from 270 to 145 ms.

What a token costs

A GPU bills by the hour, busy or not. When it decodes a batch of 10 at 14 ms a step, it makes 10 / 0.014 = 714 tokens a second, which is 2,571,429 tokens an hour. At $2.00 an hour that is $2.00 / 2.571429 = $0.78 per million output tokens. Prefill makes 8,000 prompt tokens a second, which costs $0.07 per million. If the GPU sits idle half the time, every token costs twice as much.

In your own words
In your own words: why does a prefix cache lower the time to first token but not the time of each later word?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: a server with a queue, a cache, replicas and a router

The setup above holds a small simulation. One GPU does continuous batching. Each request that gets a seat first pays its prefill, which blocks the GPU, then joins the batch. The reply has 50 to 300 tokens and the prompt has 400 to 2,000, so the average prompt is 1,200 tokens, as in the example. serve returns each request's time to first token and the tokens made. dollars_per_million turns the GPU's hourly price into a price per million tokens.

The first cell raises the arrival rate on one GPU and prints the median (p50) and the 95th percentile (p95) of the time to first token, and the price.

Worked example

One GPU as the load rises

Simulate 60 seconds at 1, 2, 3, 4 and 6 requests a second.

Setup (runs before every cell) · 41 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import numpy as np

PREFILL_RATE = 8000        # prompt tokens a second: an assumption for this lesson
PRICE = 2.00               # dollars an hour for one GPU: an assumption, not a quote

def step_time(batch):
    return 0.010 + 0.0004 * batch

def make_requests(rate, seconds=60, seed=0):
    rng = np.random.default_rng(seed)
    arrivals = np.cumsum(rng.exponential(1 / rate, int(rate * seconds * 2)))
    arrivals = arrivals[arrivals < seconds]
    prompt = rng.integers(400, 2001, len(arrivals))
    reply = rng.integers(50, 301, len(arrivals))
    return arrivals, prompt, reply

def serve(arrivals, prompt, reply, hit=0.0, max_batch=16, speed=1.0, seconds=60):
    """One GPU, continuous batching. Each admitted request pays its prefill first."""
    clock, nxt, waiting, active = 0.0, 0, [], []
    left, first, made = reply.copy(), np.full(len(arrivals), np.nan), 0
    while clock < seconds:
        while nxt < len(arrivals) and arrivals[nxt] <= clock:
            waiting.append(nxt)
            nxt += 1
        if waiting and len(active) < max_batch:
            r = waiting.pop(0)
            clock += prompt[r] * (1 - hit) / PREFILL_RATE * speed
            first[r] = clock - arrivals[r]          # time to first token
            active.append(r)
        elif active:
            clock += step_time(len(active)) * speed
            made += len(active)
            for r in active:
                left[r] -= 1
            active = [r for r in active if left[r] > 0]
        else:
            clock = arrivals[nxt] if nxt < len(arrivals) else seconds
    return first[~np.isnan(first)], made

def dollars_per_million(made, gpus=1, seconds=60, price=PRICE):
    return price * gpus * seconds / 3600 / (made / 1e6)
print(f"{'req/s':>5} {'p50 first token':>16} {'p95 first token':>16}",
      f"{'$ / million tokens':>19}")
for rate in (1, 2, 3, 4, 6):
    first, made = serve(*make_requests(rate))
    p50, p95 = np.percentile(first, [50, 95]) * 1000
    print(f"{rate:>5} {p50:>13.0f} ms {p95:>13.0f} ms",
          f"{dollars_per_million(made):>19.2f}")
In the lesson you can change this and run it.

Up to 2 requests a second the first word takes about 175 ms and a million tokens cost $1.93 to $3.58, mostly paid for idle time. At 4 a second a queue forms: the first word takes 4.7 seconds at the median. The price stops falling at about $1.05, because the GPU is full.

Quick check
In the table, the price per million tokens falls as the rate goes from 1 to 4 requests a second, then stays near $1.05. Why?
  1. The simulation stops charging after 4 requests a second
  2. The requests get shorter as the load rises
  3. Prices are lower for busy GPUs by contract with the host, which gives volume discounts at 4 a second
  4. At low load the GPU is idle much of the time but still billed; once it is full, more requests only add waiting
Show the answer

At low load the GPU is idle much of the time but still billed; once it is full, more requests only add waiting

The hourly bill does not change with load. Tokens made per hour rise until the GPU is full, then stop rising, and the price per token follows.

Quick check
A model serves 9 requests in 1 second each and 1 request in 20 seconds. What does this print: the median and the 95th percentile of the waits?
import numpy as np

waits = [1] * 9 + [20]
print(np.percentile(waits, 50), round(np.percentile(waits, 95), 2))
  1. 1.0 1.0
  2. 1.0 20.0
  3. 1.0 11.45
  4. 2.9 20.0
Show the answer

1.0 11.45

The median is the middle wait, 1.0. The 95th percentile lies between the 9th and 10th sorted values, so it is pulled most of the way toward the slow one: 11.45.

Now try four ways to run the same 4 requests a second.

  • A prefix cache that holds 60% of every prompt.
  • A second replica: two GPUs, the requests shared by turns. It costs twice the GPU-hours.
  • A router: prompts under 1,000 tokens go to a small model that is 2.5 times faster and costs $0.80 an hour. The rest go to the large one.
Worked example

Four ways to run 4 requests a second

Compare the single GPU with a prefix cache, two replicas and a small-then-large router: p50 and p95 time to first token, and the price per million tokens.

Uses the setup above, 41 lines.

arrivals, prompt, reply = make_requests(4)
def report(label, first, made, cost):
    p50, p95 = np.percentile(first, [50, 95]) * 1000
    print(f"{label:<24} {p50:>7.0f} ms {p95:>7.0f} ms {cost:>8.2f}")

print(f"{'at 4 requests a second':<24} {'p50':>10} {'p95':>10} {'$/M tokens':>10}")
first, made = serve(arrivals, prompt, reply)
report("one GPU", first, made, dollars_per_million(made))
first, made = serve(arrivals, prompt, reply, hit=0.6)
report("60% of the prompt cached", first, made, dollars_per_million(made))

even = np.arange(len(arrivals)) % 2 == 0
halves = [serve(arrivals[m], prompt[m], reply[m]) for m in (even, ~even)]
first = np.concatenate([h[0] for h in halves])
made = sum(h[1] for h in halves)
report("two replicas", first, made, dollars_per_million(made, gpus=2))

easy = prompt < 1000
small = serve(arrivals[easy], prompt[easy], reply[easy], speed=0.4)
large = serve(arrivals[~easy], prompt[~easy], reply[~easy])
first = np.concatenate([small[0], large[0]])
made = small[1] + large[1]
cost = (PRICE + 0.8) * 60 / 3600 / (made / 1e6)
report("router: small then large", first, made, cost)
In the lesson you can change this and run it.

Each fix removes the queue in a different way. The cache cuts the work, so the p95 falls from 7.6 seconds to 0.75 and the price falls too. The replicas cut the queue by adding a GPU, so the p95 is 257 ms and the price rises to $1.82. The router sends short prompts to a cheaper model, so the p95 is 520 ms at $1.29.

Plot how the p95 time to first token grows with the load, with and without the cache. The log axis keeps both curves in view.

Uses the setup above, 41 lines.

import matplotlib.pyplot as plt

rates = [1, 2, 3, 4, 6]
for label, hit in [("one GPU", 0.0), ("60% of the prompt cached", 0.6)]:
    p95 = [np.percentile(serve(*make_requests(r), hit=hit)[0], 95) for r in rates]
    plt.plot(rates, p95, marker="o", label=label)
plt.yscale("log")
plt.xlabel("requests a second")
plt.ylabel("p95 time to first token (s)")
plt.title("The first token under rising load")
plt.legend()
plt.show()
Drill

Time to first token

Write ttft(queue, prompt, cached, rate=8000): the queue (in seconds) plus the time to prefill the prompt tokens that are not cached.

Uses the setup above, 41 lines.

def ttft(queue, prompt, cached, rate=8000):
    return 0.0

print(ttft(0.12, 1200, 0), ttft(0.12, 1200, 1000))
In the lesson, your code is checked as you write it.
Drill

Dollars per million tokens

Write dollars_per_million(price_per_hour, tokens_per_second): the price of a million tokens from a GPU's hourly price and the tokens it makes each second.

Uses the setup above, 41 lines.

def dollars_per_million(price_per_hour, tokens_per_second):
    return 0.0

print(round(dollars_per_million(2.0, 714.2857), 2))
In the lesson, your code is checked as you write it.
In your own words
In your own words: why did two replicas lower the p95 but raise the price per million tokens?

Write a sentence or two, then compare it with a good answer in the lesson.

In production: capacity, the tools and the on-call view

You choose how many GPUs from the arrival rate. Each request asks for two kinds of work: prefill time on the GPU, and its share of decode time. For the simulation's mix, a request needs 1,200 / 8,000 = 0.15 s of prefill. It also needs 175 tokens of decode. A GPU decoding a batch of 16 makes 16 / 0.0164 = 975 tokens a second, so the decode takes 175 / 975 = 0.18 s of GPU time. Add them: about 0.33 GPU-seconds for every request. At 3 requests a second the GPU is nearly full, which is where the table's p95 jumped.

Worked example

How many GPUs for an arrival rate

Work out the GPU load of the request mix from the sums above, and the number of GPUs for each arrival rate.

Uses the setup above, 41 lines.

import math

prefill_s = 1200 / PREFILL_RATE                          # mean prompt, in seconds
decode_s = 175 / (16 / step_time(16))                    # mean reply at a batch of 16
per_request = prefill_s + decode_s
print(f"GPU-seconds for each request: {per_request:.2f}")
for rate in (1, 2, 3, 4, 6, 10):
    load = rate * per_request
    gpus = math.ceil(load / 0.7)
    print(f"{rate:>2} requests a second: load {load:.2f} -> {gpus} GPUs at 70% busy")
In the lesson you can change this and run it.

A GPU should not run at 100%: a burst then has nowhere to go. Planning for 70% busy, 4 requests a second needs 2 GPUs, and 10 need 5.

Quick check
Each request needs 0.33 GPU-seconds. What does this print for the GPU load of 6 requests a second?
rate, per_request = 6, 0.33
print(round(rate * per_request, 2))
  1. 6.0
  2. 1.8
  3. 1.98
  4. 0.055
Show the answer

1.98

6 requests a second × 0.33 GPU-seconds each = 1.98 seconds of GPU work every second. One GPU has one second to give, so a load above 1 means a queue that keeps growing.

Where it shows up in your week

On call for a model service, you watch four numbers: the queue length, the p95 time to first token, the prefix-cache hit rate and GPU utilization. A rising queue with a flat arrival rate points at a change in prompt length or a cache that stopped hitting. A bill that rises while traffic is flat points at low utilization. Capacity is planned from the arrival rate and the request mix, as above, with room for bursts.

Quick check
A GPU costs $2 an hour and is busy a quarter of the time. Compared with a fully busy GPU, what does each token cost?
  1. A quarter as much: fewer tokens means a smaller bill
  2. The same: the price depends only on the hourly rate
  3. Nothing extra: an idle GPU isn't charged for
  4. Four times as much: the idle hours are still billed
Show the answer

Four times as much: the idle hours are still billed

The bill is $2 an hour whatever the load. A quarter of the tokens for the same money makes each one four times dearer.

In your own words
In your own words: traffic is flat, but the p95 time to first token has doubled. What would you check first, and why?

Write a sentence or two, then compare it with a good answer in the lesson.

Goes deeper in

Practice what you just read

  • 4 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

Time to first token is mostly which two parts?

  1. The queue and the prefill
  2. The decode and the stream
  3. Tokenizing and detokenizing
  4. The weights loading and the cache building

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 6 · about 35 hours of study

Modern LLM engineering

Post-training, fine-tuning, serving and safety 48 lessons 2 free
  1. Base models vs assistants FreeWhat post-training adds to a pretrained model.30 min
  2. Supervised fine-tuningTeach behavior from example conversations.40 min
  3. Instruction and preference dataThe datasets that shape assistant behavior.30 min
  4. RLHFReward models and reinforcement learning from human feedback.40 min
  5. DPO and direct preference methodsPreference training without a separate reward model.40 min
  6. Constitutional AITraining with AI feedback guided by written principles.35 min
  7. Training reasoning modelsReinforcement learning on verifiable problems.40 min
  8. Prompting vs RAG vs fine-tuningPick the cheapest approach that works.30 min
  9. LoRAFine-tune a big model by training small low-rank updates.40 min
  10. Fine-tune an open modelRun a real fine-tune and evaluate the result.45 min
  11. DistillationTrain a small model to imitate a large one.35 min
  12. Tuning an embedding model for retrievalWhen search fails on your own vocabulary: train an adapter with a contrastive loss on question and passage pairs, test on held-out passages, use batch size and hard negatives, and measure what tuning costs elsewhere.45 min
  13. QuantizationStore weights in fewer bits to run models cheaply.35 min
  14. Quantization in practice: zero points, calibration and formatsScale and zero point, per-channel scales, calibration sets, error on a layer's output, quantization-aware training and the file formats you meet at work.35 min
  15. Serving modelsBatching, throughput and latency in production.35 min
  16. One request through a serving stack: latency and costFollow one request through queue, prefill, a prefix cache, decode and the stream, and price a token from a GPU's hourly cost.35 min
  17. Speculative decodingDraft with a small model, verify with a big one.35 min
  18. Running open models locallyDownload, quantize and run a model on your own machine.30 min
  19. Will it fit? Memory for serving and fine-tuningWeights, KV cache, runtime and adapter states added up for a model card and a GPU, with a calculator.30 min
  20. InterpretabilityLooking inside models to see what they compute.40 min
  21. Confident and wrongWhy a model answers with the same assurance whether it is right, wrong or asked about something that does not exist.30 min
  22. Why it can't spell or count lettersThe model sees token IDs, and an embedding learned from usage carries nothing about spelling.25 min
  23. Copying from the promptIn-context learning as a two-head copying circuit: the base tiny GPT cannot copy a repeated pattern, a model trained on repeats grows an induction head and can.30 min
  24. Why temperature changes the answerThe same prompt at 0, 0.7 and 1.5: deterministic loops, plausible variety, broken grammar, measured on a real model.25 min
  25. Why long prompts diluteAttention is a budget shared by every token, so a fact far from the question loses its share; measured, with the cliff beyond the trained window.25 min
  26. Why it never says "I don't know"A chat model answers every request because its fine-tuning data always did; preference training adds flattery; how to build abstention in.25 min
  27. Benchmarks and their limitsWhat leaderboards measure, and what they miss.38 min
  28. Alignment and safetyMaking capable models behave as intended.35 min
  29. Evaluating agentsMeasure multi-step, tool-using behavior.35 min
  30. Reading AI papersGet the idea out of a research paper in 30 minutes.30 min
  31. Capstone: a tuned, evaluated assistantFine-tune, evaluate and deploy a specialist assistant.45 min
  32. Weight-space programming: what it is, and what has been shown Free OptionalA language model reads and writes another network's weights with no training loop: the published experiments, their limits and the criticism, and one edit you make by hand.30 min
  33. Anatomy of a checkpoint OptionalA model file is a set of named tensors: count them, find the ones that write to the residual stream, and save one to bytes and back.35 min
  34. Reading weights without running the model OptionalSingular values, and an attention head's token-to-token circuit, read from the matrices alone and then checked against a measurement.35 min
  35. Writing weights by hand OptionalSet a small network's 55 weights so it catches every ball in a paddle game, the way a program would be written, and meet compiled transformers.40 min
  36. Edits are deltas: task vectors, patches and hashes OptionalSubtract a base from a fine-tune and the edit becomes an object: scale it, take it away, ship it as a patch, and prove by hash what was applied.35 min
  37. Merging models OptionalAverage two fine-tunes, add their task vectors, try SLERP, TIES and DARE, measure the interference, and compare with a model trained on both tasks.40 min
  38. Low-rank edits OptionalCut a fine-tune's change down to its largest directions with the SVD: how few a behavior needs, and how to pull an adapter out of a delta.35 min
  39. Directions: steer the run or edit the weights OptionalFind a behavior's direction in the residual stream, add it at run time, project it out of the weights, and see the effect change from input to input.40 min
  40. Editing one fact OptionalA rank-one change to one matrix rewrites what a product is made from; then reliability, generalization, locality, and what breaks after many edits.40 min
  41. The editor's harness OptionalWhat an LLM that edits weights is given and what it is denied: weights as text and their cost in tokens, a sandbox, a budget of forward passes, and rules the harness enforces.40 min
  42. An LLM reads a network's weights OptionalSix recorded sessions: an LLM is handed 1,315 weights as text and asked what the network was trained to do. Score each answer against the network, and study one it got wrong.40 min
  43. An LLM fixes a model's weights OptionalFind a tiny GPT's loop by hand, then replay recorded sessions in which an LLM hunts for it with ten forward passes, and measure every fix on held-out questions.45 min
  44. An LLM writes a network's weights OptionalFive recorded sessions: an LLM writes every weight of a paddle network from the game's source and one sentence. Score each against its specification, test its design claims, and edit one by hand.40 min
  45. Did it work? Testing an edit OptionalReview a patch for a tiny GPT that can't stop talking: held-out questions, a paired comparison, a same-size control, and what the edit cost.40 min
  46. The published edit, on a real model OptionalRead the edit published for an open model of 870 million weights, apply and verify one in the same form, see what 16-bit and 4-bit storage do to a fix, and run the real thing from a notebook.45 min
  47. Safety, licenses and provenance OptionalBefore a custom model is released: trace its lineage by hash, work out what its parents' licenses pass on, see what an edit can do to a safeguard, and write its card.40 min
  48. Capstone: ship a custom model OptionalSet release gates, then diagnose, edit, measure and document a tiny GPT with a fault no lesson has fixed, and write the record a stranger could check.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write