Index Ai Academy
Glossary · Phase 6 · Modern LLM engineering

Prefix cache

Saved keys and values for a prompt start that many requests share, so the server skips that work.

Try it now

The first token is out at 270 ms. The prefill's cost is 150 ms of a whole GPU. The decode's cost is shared by 10 requests.

Learn it properly

One request through a serving stack: latency and cost · Follow one request through queue, prefill, a prefix cache, decode and the stream, and price a token from a GPU's hourly cost. About 35 minutes.

Other terms from the same lesson

  • prefill: The first stage of answering: the model reads the whole prompt in one pass and fills its key-value cache.
  • replica: One of several identical copies of a service running side by side to share the traffic.
Keep learning

More in plain English.

1286 terms in the glossary, each with the lesson that teaches it.

1286 terms

The glossary

Every word you'll meet in AI and machine learning, explained in everyday language.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write