Serving models
Batching, throughput and latency in production.
Create a free account Try part of it here
Why it matters
Training a model happens once. Serving it, which means answering people's requests, goes on every day for as long as the model is in use. How a provider groups requests, uses GPU memory and routes traffic decides three things you notice. It sets what you pay per token. It sets how fast answers stream. And it explains why work that can wait costs less. If you ever host an open model yourself, these are the choices you'll make.
What you'll be able to do
- Explain why batching is essential for serving LLMs, and how continuous batching works
- Trade off throughput against per-user speed with a simple cost model
- Estimate how many conversations fit on a GPU, and why paged KV caches help
- Name the metrics and tools of production serving (TTFT, tokens/s, p95, vLLM)
What it covers
- One request at a time wastes the GPU
- A simulation you can run
- Throughput versus speed for each user
- Memory decides how big the batch can be
- The production toolkit
- Worked example: picking a batch size for a chat app
One request at a time wastes the GPU
Picture a pizza oven. Baking one pizza takes about as long as baking eight side by side, because heating the oven is the slow part. A shop that bakes one pizza at a time wastes most of its oven.
A GPU serving a language model works the same way. You know from the KV cache lesson that a model writes its answer one token at a time. (A token is a word or a piece of a word.) Each new token takes one decode step: one pass through the model. In every step, the GPU must read all of the model's weights from its memory. That reading is the slow part, like heating the oven.
Once the weights are read, using them for one conversation or for twenty costs almost the same. So servers batch. Each decode step writes the next token for many conversations at once. The group of conversations is the batch, and how many are in it is the batch size.
Two ways to batch
How you batch matters a lot.
- Static batching waits for a group of requests, say 8. It runs them together until the longest answer is done. Short answers finish early, but their slots sit empty. New requests wait outside until the whole group is finished.
- Continuous batching checks after every single step. Finished requests leave at once, and waiting requests join at once. It came from a research system called Orca in 2022. Today it's standard.
Think of a restaurant. With static batching, the host seats a table of 8 and lets nobody else sit down until all 8 have finished dessert. With continuous batching, the next person in line takes a chair as soon as anyone leaves.
Show the answer
Everyone waits for the longest answer, so finished slots sit idle
Continuous batching fills each slot as soon as it frees up.
A simulation you can run
The code below pretends to be one GPU for two minutes. Requests arrive at random, about 4 per second. Each one needs a reply between 50 and 500 tokens long. A decode step takes 10 ms (a ms, or millisecond, is a thousandth of a second). It also takes 0.4 ms more for each conversation in the batch.
The code tries three policies: one request at a time, static batches of 8, and continuous batching with up to 32 conversations. For each one it prints five numbers:
- tokens/s: tokens made per second, for all users together. This is the throughput.
- finished: how many answers were completed.
- median wait: the middle wait time. Half the requests waited less and half waited more. A wait runs from the moment a request arrives until its last token.
- p95 wait: the 95th percentile. 95% of requests waited less than this. It shows how bad things get for the unlucky few.
- still queued: requests still waiting to start when the clock stops.
You don't need to follow every line. The comments say what each part does.
import numpy as np rng = np.random.default_rng(0) RATE, SECONDS = 4, 120 # 4 requests per second, for 2 minutes arrivals = np.cumsum(rng.exponential(1 / RATE, 2 * RATE * SECONDS)) arrivals = arrivals[arrivals < SECONDS] lengths = rng.integers(50, 501, len(arrivals)) # tokens each reply needs def step_time(batch): # one decode step, in seconds: mostly reading return 0.010 + 0.0004 * batch # the weights, plus a little per sequence def simulate(policy, max_batch=32, static_size=8): clock, arrived, waiting, active, latencies, tokens = 0.0, 0, [], [], [], 0 left = lengths.copy() while clock < SECONDS: while arrived < len(arrivals) and arrivals[arrived] <= clock: waiting.append(arrived) arrived += 1 if policy == "one at a time" and not active and waiting: active = [waiting.pop(0)] elif policy == "static batches" and not active and len(waiting) >= static_size: active, waiting = waiting[:static_size], waiting[static_size:] elif policy == "continuous batching": while waiting and len(active) < max_batch: # fill free slots every step active.append(waiting.pop(0)) if not active: # idle until the next request arrives clock = arrivals[arrived] if arrived < len(arrivals) else SECONDS continue clock += step_time(len(active)) tokens += len(active) for r in active: left[r] -= 1 latencies += [clock - arrivals[r] for r in active if left[r] == 0] active = [r for r in active if left[r] > 0] lat = np.array(latencies) return tokens / SECONDS, len(lat), np.median(lat), np.percentile(lat, 95), len(waiting) print(f"{len(arrivals)} requests arrive over {SECONDS} s\n") print(f"{'policy':<20} {'tokens/s':>8} {'finished':>9} {'median wait':>12} {'p95 wait':>9} {'still queued':>13}") for policy in ["one at a time", "static batches", "continuous batching"]: tput, n, med, p95, queued = simulate(policy) print(f"{policy:<20} {tput:>8.0f} {n:>9} {med:>11.1f}s {p95:>8.1f}s {queued:>13}")
With one request at a time, the GPU falls far behind. It finishes 46 answers in two minutes and leaves a queue of hundreds. Static batches help, but slots sit idle while each batch waits for its longest answer.
Continuous batching gives about ten times the throughput of one at a time. It keeps up with the traffic, so nothing is left queued. (The requests not yet finished are the ones still being answered when the clock stops.) Its median wait stays under 5 seconds. It's the same GPU and the same model. Only the scheduling changed.
Show the answer
Lets new requests join, and finished ones leave, after every decode step
No slot sits idle waiting for the batch's longest answer. The batch is refilled step by step.
Throughput versus speed for each user
A bigger batch makes more tokens in total. But each step takes a little longer, so each user's words arrive a little slower. A city bus works the same way: more stops carry more riders, but every rider's trip gets longer.
Here is the rule this lesson uses for one step. It's a simple model, not a measurement of a real GPU, but it has the right shape.
Every step gives each user in the batch one token. So two speeds follow from the step time, measured in seconds:
In plain words: each step costs a fixed 10 ms, plus a little for every conversation in the batch. Each user gets one token per step. The whole GPU makes one token per conversation per step.
Try a batch of 8. One step takes 10 + 0.4 × 8 = 13.2 ms, which is 0.0132 seconds. Each user gets 1 ÷ 0.0132 ≈ 76 tokens per second. The whole GPU makes 8 ÷ 0.0132 ≈ 606 tokens per second. The code below does the same sum for four batch sizes:
def step_time(batch): return 0.010 + 0.0004 * batch print(f"{'batch':>5} {'total tokens/s':>15} {'each user sees':>15}") for batch in [1, 8, 32, 128]: t = step_time(batch) print(f"{batch:>5} {batch / t:>15,.0f} {1 / t:>12.0f} tokens/s")
Read the table from top to bottom. A batch of 128 makes about 22 times as many tokens per second as a batch of 1: 2,092 against 96. But each user's stream drops from 96 to 16 tokens per second.
Providers pick a point on this trade-off. Chat needs each user's stream to stay readable, often 30 or more tokens per second. It also needs a short time to first token (TTFT): the wait before the first word appears. TTFT is mostly time spent in the queue plus prefill, the pass where the model reads your prompt.
Jobs that run in the background don't need speed per user, so they can use huge batches. That's why APIs offer batch endpoints at a discount. Major providers charge about half price for requests you're willing to wait up to a day for. They run them in large batches when GPUs are free.
This is the simple model from this section. Bigger batches slow each user a little and raise the total a lot.
batch = 10 step = 0.010 + 0.0004 * batch print(round(batch / step), round(1 / step))
Show the answer
714 71
The step takes 10 + 0.4 × 10 = 14 ms, or 0.014 s. Each user gets 1 ÷ 0.014 ≈ 71 tokens per second. Ten users together get 10 ÷ 0.014 ≈ 714.
Memory decides how big the batch can be
Every conversation in the batch needs its own KV cache. That's the model's saved keys and values for every token so far, from the KV cache lesson. The cache lives in GPU memory, next to the weights.
Think of GPU memory as a desk. The weights take up a fixed part of it. Each conversation needs its own pile of notes on the space that's left. When the desk is full, nobody new can sit down. So the batch can only be as big as the number of piles that fit:
In plain words: take the GPU's memory and subtract what the weights need. Divide what's left by the space one conversation needs. Round down, since half a conversation is no use.
Here's a worked example. An 80 GB GPU serves Llama 3 8B, a model with 8 billion weights. (A GB, or gigabyte, is a billion bytes.) Stored in 16-bit numbers, each weight takes 2 bytes, so the weights fill 16 GB. That leaves 64 GB. Each token of each conversation needs 128 KiB of cache, which is 131,072 bytes. (A KiB is 1,024 bytes.)
- Reserve room for 8,192 tokens per conversation: 131,072 × 8,192 ≈ 1.074 GB each. 64 ÷ 1.074 ≈ 59.6, so 59 conversations fit.
- Hand out memory as tokens arrive, with 2,000 tokens used on average: 131,072 × 2,000 ≈ 0.262 GB each. 64 ÷ 0.262 ≈ 244.1, so 244 conversations fit.
Go deeper Where 128 KiB per token comes from
The code below writes the cache per token as 2 × 32 × 8 × 128 × 2. This is the KV cache formula from the earlier lesson, read from left to right:
- 2: the cache keeps a key and a value for each token.
- 32: Llama 3 8B has 32 layers, and each layer keeps its own.
- 8: each layer has 8 KV heads.
- 128: each head stores 128 numbers.
- 2: each number takes 2 bytes in 16-bit format.
Multiplied out, that's 131,072 bytes, or 128 KiB.
The same sums in Python:
gpu, weights = 80e9, 16e9 # an 80 GB GPU serving an 8B model in 16-bit per_token = 2 * 32 * 8 * 128 * 2 # Llama 3 8B: 128 KiB of KV cache per token for policy, reserved in [("reserve the max context (8,192 tokens) per request", 8192), ("allocate as tokens arrive (average 2,000 used)", 2000)]: print(f"{policy:<52} {int((gpu - weights) // (per_token * reserved)):>4} conversations")
It prints 59 and 244. Same GPU, same model, and four times as many conversations.
Paging: hand out memory only when it's needed
Older servers reserved memory for the longest possible reply up front. Most of it went unused. It's like a restaurant that holds a table for twelve for every party, even when two people show up.
PagedAttention fixed this. It came from the vLLM project in 2023. It stores the KV cache in small blocks of a fixed size. A conversation gets a new block only when it needs one, as tokens are generated. Operating systems manage a computer's memory the same way, in blocks called pages. That's where the name comes from.
The vLLM authors measured that earlier systems wasted 60–80% of KV cache memory. Paging cut that waste to almost nothing. More conversations in memory means bigger batches. Bigger batches mean more throughput from the same hardware. Quantizing the weights or the cache, from the previous lesson, stretches memory further still.
gpu, weights, per_conversation = 24, 8, 0.5 print(int((gpu - weights) // per_conversation))
Show the answer
32
24 − 8 leaves 16 GB for caches. 16 ÷ 0.5 = 32 conversations. Dividing the whole 24 GB would forget that the weights take space too.
The production toolkit
Teams that serve models in production watch a few numbers and use a few standard tools.
- Metrics. TTFT, time per output token, and throughput. Also tail latency: the p95 and p99, the times that 95% and 99% of requests beat. They're called the tail because the slowest requests sit at the end, or tail, of the sorted list of times. Teams also track cost per million tokens and GPU utilization, the share of time the GPU is busy.
- Serving engines. vLLM, SGLang, TensorRT-LLM and Hugging Face TGI come with continuous batching, paged KV caches, quantization and prefix caching built in.
- Advanced tricks. Chunked prefill splits a long prompt into pieces, so it doesn't stall everyone else's decoding. Some teams put prefill and decode on different GPUs. Prefix caching reuses work on a prompt start that many requests share; it's the server side of prompt caching. Speculative decoding is the next lesson.
- Routing. Easy requests go to small models and hard ones go to large models. Autoscaling adds copies of the model, called replicas, as traffic rises, and removes them as it falls.
Why watch the tail and not only the average? Say 100 requests come in. 90 take 2 seconds and 10 take 30 seconds. The average is (90 × 2 + 10 × 30) ÷ 100 = 4.8 seconds, which sounds fine. But the p95 is 30 seconds, because one user in ten waits half a minute. Those are the users who complain.
For marketers: if a job doesn't need answers right away, use the provider's batch API. Tagging last month's reviews and writing next week's product copy are good examples. You get the same model at roughly half the price.
Show the answer
The average is about 3 seconds, but the p95 is 20 seconds
The average is (94 × 2 + 6 × 20) ÷ 100 = 3.08 seconds. Sort the times: the first 94 are 2 seconds, so the 95th is already a slow one. The p95 is 20 seconds, and the average hides it.
Worked example: picking a batch size for a chat app
You run a chat app with the step-time rule from this lesson: 10 ms plus 0.4 ms per conversation. You want every user to get at least 40 tokens per second. The code tries four batch sizes and prints the step time, each user's speed and the total. Before you run it, guess which batch sizes meet the target.
base_ms, per_seq_ms = 10, 0.4 # one decode step: 10 ms + 0.4 ms per conversation for batch in [10, 20, 30, 40]: step_ms = base_ms + per_seq_ms * batch each_user = 1000 / step_ms # tokens per second for one user total = batch * each_user # tokens per second for the whole GPU print(f"batch {batch:>2}: step {step_ms:.0f} ms, each user {each_user:.0f} tokens/s, total {total:,.0f} tokens/s")
A batch of 10 gives each user 71 tokens per second, but the GPU makes only 714 in total. At 20, each user gets 56 and the total rises to 1,111. At 30, each user still gets 45, above the target, and the total is 1,364. At 40, each user drops to 38, below 40. So of these four, 30 is the best choice: almost double the throughput of a batch of 10, and every user still reads comfortably. Memory isn't the limit here. With paging, the 80 GB GPU from earlier had room for 244 conversations.
Practice what you just read
- 4 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 5 review cards for spaced repetition
Try a quiz question
Why does batching help LLM serving so much?