Index Ai Academy
Phase 5 · Beyond text

The voice agent loop and its latency budget

One turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.

About 45 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers8

Why it matters

A voice agent is a chat agent with ears and a mouth: recognition in front of the model, synthesis after it. Wiring the three together takes an afternoon. Making the result pleasant to talk to is a different job, and almost all of it is about time. People answer each other within a fraction of a second, notice a pause of one second, and give up on a system that leaves three. They also interrupt, and stop to think in the middle of a sentence. This lesson is about the loop that copes with all of that, and the arithmetic of where its milliseconds go.

What you'll be able to do

  • Describe one turn of a voice agent: listening, deciding the person has finished, thinking, speaking
  • Add up a latency budget, and show why streaming every stage changes it
  • Explain the trade in deciding when someone has stopped talking, and how to improve on a fixed wait
  • Handle an interruption: stop speaking, and record only what was heard
  • Read a turn's log, find the slowest stage, and judge latency by its slow turns as well as its typical ones
Quick check
A guess before you start. A voice agent recognizes speech in 0.4 seconds, its model replies at 50 tokens a second, and its synthesizer makes audio four times faster than it plays. Each stage waits for the one before to finish completely. Where will most of the silence before a 16-second spoken reply come from?
  1. The model writing the first token of its reply
  2. The loudspeaker starting up before it can play
  3. Recognizing what the person said, at 0.4 seconds
  4. Synthesizing the whole reply before playing it
Show the answer

Synthesizing the whole reply before playing it

Sixteen seconds of speech made four times faster than real time still takes four seconds to make. If nothing is played until all of it exists, those four seconds are silence, far more than recognition or the model's first token. The cure is not a faster synthesizer. It is to start playing the first sentence while the rest is still being made.

The loop: one turn, and where the time goes

This rests on Recognition and synthesis as a pipeline, the stage that turns audio into words, and on Streaming responses, which sends a model's reply token by token.

One turn

One turn of a voice agent has four parts.

  1. Listen. Audio streams in from the microphone.
  2. Decide the person has finished. Called endpointing: the agent waits for a stretch of silence.
  3. Think. The words go to the model, as in any chat.
  4. Speak. The reply is synthesized and played.

What the person experiences is one number: the silence between the end of their sentence and the start of the agent's voice. The cell adds it up for two designs. The stage times are assumptions for an ordinary setup, chosen to compare the designs.

# One turn of a voice agent, in milliseconds. The numbers are assumptions for a typical
# setup, chosen to compare two designs.
TOKENS = 52                      # the reply as the model writes it: about 40 words
TOKENS_PER_SECOND = 50
FIRST_SENTENCE_TOKENS = 11       # "Sure, I can move that to Thursday." or the like

# how long the agent waits, after the last sound, before
# deciding the person has finished
wait_for_silence = 700
one_at_a_time = [
    ("wait for silence", wait_for_silence),
    ("recognize the whole recording", 400),
    ("model: first token", 400),
    ("model: the rest of the reply", round(1000 * TOKENS / TOKENS_PER_SECOND)),
    # 16 seconds of speech, made four times faster than it plays
    ("synthesize the whole reply", 4000),
]
streaming = [
    ("wait for silence", wait_for_silence),
    ("finish recognizing (it ran while they spoke)", 100),
    ("model: first token", 400),
    ("model: the first sentence",
     round(1000 * FIRST_SENTENCE_TOKENS / TOKENS_PER_SECOND)),
    ("synthesize the first sentence", 150),
]
for name, stages in (("each stage waits for the one before to finish", one_at_a_time),
                     ("every stage streams", streaming)):
    total = sum(ms for stage, ms in stages)
    print(name + ": silence before the agent starts to speak =", total, "ms")
    for stage, ms in stages:
        print("   ", stage.ljust(46), str(ms).rjust(5), "ms",
              str(round(100 * ms / total)).rjust(3) + "%")
It runs right here in your browser, with nothing to install.

With each stage waiting for the last to finish, the silence is 6,540 ms, and 61% of it is synthesis: the whole reply is made before any of it plays.

Streaming changes what each stage waits for. Recognition runs while the person talks, so only a little is left to finish at the end. The model's tokens are passed on as they appear. The synthesizer starts on the first sentence as soon as it is complete, and playback begins while later sentences are still being written. The silence drops to 1,570 ms.

The model is not one millisecond faster. The total work is the same. Only the time to the first sound has changed, and that is the number people feel.

Look at what is now the largest item: 45% of the streaming budget is spent waiting to be sure the person has stopped. The next part is about that wait.

Quick check
From the lesson on streaming responses: why does a chat app show a model's reply token by token?
  1. Streaming makes the model generate its tokens faster
  2. The reader can start reading long before the reply is finished
  3. Tokens are cheaper when they are sent one at a time
  4. The model needs to see its reply on screen to continue
Show the answer

The reader can start reading long before the reply is finished

Streaming doesn't speed the model up. It lets the person start reading as soon as the first words exist, so the wait they feel is the time to the first token and not the time to the last. A voice agent applies the same idea at every stage, and the stakes are higher: unread text on a screen is fine, and silence in a conversation is not.

Try it now
In your own words
Streaming cut the silence from 6,540 ms to 1,570 ms without making any stage faster. How?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: endpoints and interruptions

Have they finished?

A chat app knows when you are done: you press Enter. A voice agent has to guess, from silence. And people fall silent in the middle of sentences all the time: "Can you move my appointment to... Thursday."

So endpointing is a wait: if nobody has spoken for this long, the turn is over. Too short, and the agent talks over people who were only thinking. Too long, and every turn starts with dead air. The cell simulates 2,000 things people say, each with pauses inside it, and tries five waits. Then it tries a smarter rule.

import random

def utterance(rng):
    """The pauses inside one thing a person says, in ms. People stop to think in
    the middle of sentences."""
    return [rng.lognormvariate(5.3, 0.7) for i in range(rng.randint(1, 5))]

rng = random.Random(0)
talk = [utterance(rng) for i in range(2000)]
longest = sorted(max(p) for p in talk)
print("longest pause inside an utterance: half are under", round(longest[1000]),
      "ms, and one in ten is over", round(longest[1800]), "ms")

print()
print("wait this long | cut the person off | silence added to every turn")
for wait in (200, 400, 700, 1000, 1500):
    cut = sum(max(p) >= wait for p in talk) / len(talk)
    print(str(wait).rjust(11), "ms |", f"{100 * cut:5.1f}% of turns".rjust(18), "|",
          wait, "ms")

# A smarter rule: read the words so far. If they sound finished, wait a little. If
# they don't, wait longer.
rng = random.Random(1)
cut = waited = 0
for pauses in talk:
    # at each pause inside the utterance the sentence is NOT finished; the reader gets
    # that right 9 times in 10
    cut += any(p >= (1500 if rng.random() < 0.9 else 300) for p in pauses)
    # at the real end the sentence IS finished; the reader gets that right 9 times in 10
    waited += 300 if rng.random() < 0.9 else 1500
print()
print("read the words too |", f"{100 * cut / len(talk):5.1f}% of turns".rjust(18), "|",
      round(waited / len(talk)), "ms on average")

There is no good fixed wait. At 400 ms the agent cuts into 38.7% of turns. At 1,000 ms it interrupts only 2.8%, and adds a full second of silence to every turn, which is most of a comfortable budget gone before the model has seen a word.

The last line uses something silence doesn't have: the words. "Can you move my appointment to" is plainly unfinished, and "to Thursday" plainly is. A small model that reads the transcript so far, and waits 300 ms if it sounds complete and 1,500 ms if it doesn't, cuts off 9.2% of turns while adding 417 ms on average. That beats the fixed wait of 700 ms on both counts, with a reader that is wrong one time in ten.

This is why voice systems have a turn detector that looks at more than loudness: the words, and in good ones the melody of the voice, which falls at the end of a statement. A separate, simpler piece, voice activity detection, only answers "is anyone speaking right now?".

Quick check
A voice agent keeps interrupting people who pause to think. The team raises the endpointing wait from 400 ms to 1,200 ms. What is the cost?
  1. The synthesizer has less time to prepare its audio
  2. The recognizer becomes less accurate on long sentences
  3. Every turn, including quick ones, starts 800 ms later
  4. The model's replies become longer and slower
Show the answer

Every turn, including quick ones, starts 800 ms later

The wait is paid on every turn, since the agent can't know in advance which silences are endings. A longer wait fixes the interruptions and makes the whole conversation feel sluggish, even for "yes" and "no". The better fix decides from the words whether a short or a long wait is right for this particular pause.

Drill

Silence before the first sound

Write first_audio_ms(wait_ms, finish_asr_ms, first_token_ms, tokens_per_second, first_sentence_tokens, tts_ms): the silence, in milliseconds, between the person finishing and the agent's first audio in a fully streaming design. It is the wait for silence, the time to finish recognizing, the model's time to its first token, the time to write the first sentence at tokens_per_second, and the synthesizer's time for that sentence. Round the result to a whole number.

def first_audio_ms(wait_ms, finish_asr_ms, first_token_ms, tokens_per_second,
                   first_sentence_tokens, tts_ms):
    return 0


print(first_audio_ms(700, 100, 400, 50, 11, 150))
In the lesson, your code is checked as you write it.
Drill

When does the turn end?

A voice activity detector gives one value for each 20 ms frame: True if someone is speaking. Write endpoint(frames, quiet_needed): the index of the frame at which the turn is declared over. That is the first frame that completes a run of quiet_needed silent frames in a row after speech has started. Silence before the person first speaks doesn't count. Return None if the turn never ends.

def endpoint(frames, quiet_needed):
    return None


frames = [False, False, True, True, True, False, False, True, True, False, False, False,
          False]
print(endpoint(frames, 3))
In the lesson, your code is checked as you write it.

Being interrupted

People talk over each other, and a voice agent that keeps reading out its answer while you say "no, wait" is unusable. Handling an interruption, called barge-in, has three parts.

  1. Notice. Keep listening while speaking. That needs echo cancellation, or the agent hears its own voice from the loudspeaker and interrupts itself.
  2. Stop. Cut the audio at once, and cancel the reply still being generated and synthesized.
  3. Remember correctly. The conversation history must hold what the person heard, which is less than what the model wrote.

The cell walks through one interrupted turn as a small state machine.

reply = ("Your order shipped on Monday and should arrive Thursday. I can also send the "
         "tracking link by email, " "or change the delivery address if you will be "
                                    "away. Would you like me to "
                                    "do either of those?").split()
WORDS_PER_SECOND = 2.5

state, log = "idle", []
def move(ms, new, why):
    global state
    log.append((ms, state, new, why))
    state = new

move(0, "listening", "the person starts to speak")
move(2100, "thinking", "end of their turn detected")
move(3200, "speaking", "first audio of the reply")
speech_started = 3200
barge_in_heard = 3200 + 4800  # 4.8 s into the reply, the person starts talking over it
stopped = barge_in_heard + 200  # the agent needs 200 ms to notice and stop the audio
move(stopped, "listening", "the person interrupted")

for ms, before, after, why in log:
    print(str(ms).rjust(5), "ms |", before.ljust(9), "->", after.ljust(9), "|", why)

heard = int((stopped - speech_started) / 1000 * WORDS_PER_SECOND)
print()
print("the reply had", len(reply), "words. The person heard", heard, "of them:")
print("  heard    :", " ".join(reply[:heard]) + " ...")
print("  not heard:", "... " + " ".join(reply[heard:]))
print("what goes in the conversation history: the", heard,
      "words heard, marked as interrupted. Not all", len(reply))

The agent moved from listening to thinking to speaking, and back to listening when the person spoke. The interruption came 4.8 seconds into the reply, and the audio stopped 200 ms after that.

The model wrote 37 words. The person heard 12. If the history records all 37, then on the next turn the model believes it offered to send a tracking link and to change the address. The person heard neither offer. When they say "yes please", the model will act on an offer that, for them, was never made.

So the history is cut to the words that were actually played, and marked as interrupted. Getting this right needs the synthesizer to report how far playback got. It is the bug that most separates a demo from a product.

Quick check
A reply is being spoken at 2.5 words a second. It starts at 3,200 ms and is cut off at 8,200 ms. What does this print: the whole words the person heard?
started, stopped, words_per_second = 3200, 8200, 2.5
print(int((stopped - started) / 1000 * words_per_second))
  1. 12
  2. 37
  3. 20
  4. 5
Show the answer

12

The audio played for 5,000 ms, which is 5 seconds, and at 2.5 words a second that is 12.5 words: 12 whole ones. Those 12 are what the conversation history should hold for this turn, whatever the model had planned to say.

In your own words
After an interruption, why must the conversation history hold only the words that were played?

Write a sentence or two, then compare it with a good answer in the lesson.

In production: the slow turns

Budget for the slow turns

A latency budget written with one number per stage describes a typical turn. People don't judge an agent by its typical turn. They remember the one where it sat silent for three seconds.

The cell runs 5,000 streaming turns in which every stage's time varies, as real ones do. The spread given to each stage is an assumption. The model's first token is given the widest, as is usual when a model is shared and busy.

import random

rng = random.Random(0)
def turn():
    """One streaming turn. Each stage's time varies from turn to turn, and the model's
    first token varies most."""
    return {"wait for silence": 300 + rng.expovariate(1 / 60),
            "finish recognizing": rng.lognormvariate(4.5, 0.3),
            "model: first token": rng.lognormvariate(5.8, 0.6),
            "model: first sentence": rng.lognormvariate(5.3, 0.3),
            "synthesize it": rng.lognormvariate(5.0, 0.3)}

turns = [turn() for i in range(5000)]
totals = sorted(sum(t.values()) for t in turns)
print("silence before the agent speaks | typical turn:", round(totals[2500]),
      "ms | 1 turn in 20 is slower than:", round(totals[4750]), "ms | 1 in 100:",
      round(totals[4950]), "ms")
print("turns that keep within 1,000 ms:",
      f"{100 * sum(t <= 1000 for t in totals) / 5000:.1f}%", "| within 1,500 ms:",
      f"{100 * sum(t <= 1500 for t in totals) / 5000:.1f}%")

slow = [t for t in turns if sum(t.values()) > totals[4750]]
print()
print("stage".ljust(22), "| typical turn | the slowest 1 turn in 20")
for stage in turns[0]:
    typical = sorted(t[stage] for t in turns)[2500]
    print(stage.ljust(22), "|", str(round(typical)).rjust(9), "ms |",
          str(round(sum(t[stage] for t in slow) / len(slow))).rjust(6), "ms")

A typical turn has 1,162 ms of silence. One turn in 20 has more than 1,748 ms, and one in 100 more than 2,161. In a conversation of 20 turns, the person will most likely meet one of those slow turns.

The second table shows where a slow turn's time went. Four of the five stages are about as fast in the slowest turns as in typical ones. One isn't: the model's first token, at 333 ms typically and 1,147 ms in the slow turns. The tail of the whole system is the tail of one stage.

That tells you what to do. Averages are improved by shaving every stage. Tails are improved by attacking the one stage with the spread: a smaller or less loaded model, a shorter prompt, a cached prefix, or a second request raced against the first. Some agents say a short filler, "let me check", the moment a turn ends. It doesn't make anything faster, and it turns silence into speech.

One more design is worth knowing. Speech-to-speech models take audio in and give audio out with one network, with no text in between. They can be faster and can hear tone of voice. They are also harder to inspect, since there is no transcript in the middle to read, test or filter.

Quick check
A team reports that their voice agent's average response time is 900 ms. Why should you also ask for the 95th percentile?
  1. The average is always lower than the true typical time
  2. A few very slow turns can hide behind a good average
  3. The 95th percentile shows the fastest turns
  4. Percentiles are easier to compute than averages
Show the answer

A few very slow turns can hide behind a good average

An average of 900 ms can come from most turns at 700 ms and one in twenty at 3 seconds. In a conversation of twenty turns the person will probably hit one of those, and it is the one they remember. The 95th percentile says how bad that turn is: 95% of turns are faster.

Worked example

Worked example: read one turn's log

Every voice system worth running writes a timestamp at each handoff. Here is one turn's log, and the few lines that turn it into a latency budget. The person asked to move an appointment, and interrupted the reply partway through.

events = [  # one turn of a voice agent, as its log recorded it
    {"t": 0,    "event": "user_speech_start"},
    {"t": 1840, "event": "user_speech_end"},
    {"t": 2140, "event": "endpoint"},
    {"t": 2230, "event": "asr_final",
     "text": "can you move my appointment to thursday"},
    {"t": 2780, "event": "llm_first_token"},
    {"t": 3010, "event": "llm_first_sentence",
     "text": "Sure, I can move it to Thursday."},
    {"t": 3170, "event": "tts_first_audio"},
    {"t": 5900, "event": "barge_in"},
    {"t": 6080, "event": "tts_stopped"},
]
at = {e["event"]: e["t"] for e in events}
stages = [("wait for silence", "user_speech_end", "endpoint"),
          ("finish recognizing", "endpoint", "asr_final"),
          ("model: first token", "asr_final", "llm_first_token"),
          ("model: first sentence", "llm_first_token", "llm_first_sentence"),
          ("synthesize it", "llm_first_sentence", "tts_first_audio")]
total = at["tts_first_audio"] - at["user_speech_end"]
print("the person stopped at", at["user_speech_end"], "ms and the agent's voice",
      "started at", at["tts_first_audio"], "ms: a silence of", total, "ms")
for name, start, end in stages:
    ms = at[end] - at[start]
    print("   ", name.ljust(22), str(ms).rjust(4), "ms", "#" * round(ms / 25))
print("the largest share:", max(stages, key=lambda s: at[s[2]] - at[s[1]])[0])
print("the person interrupted", at["barge_in"] - at["tts_first_audio"], "ms into the",
      "reply, and the audio stopped", at["tts_stopped"] - at["barge_in"], "ms later")
In the lesson you can change this and run it.

The person stopped speaking at 1,840 ms and heard the agent at 3,170: a silence of 1,330 ms. The breakdown says where it went. The wait for silence was 300 ms, short, so this system is probably using the words to decide. Finishing recognition took 90 ms, which means it was streaming all along. The model's first token took 550 ms, the largest share and well above a typical 333, so this was one of the slower turns and the model is the stage to look at. The first sentence took another 230 ms and its synthesis 160. Then the interruption: 2,730 ms into the reply, with the audio stopping 180 ms later. That last number is a budget of its own. Much over a quarter of a second, and the agent seems not to be listening. Five subtractions turned a vague "it felt slow" into the name of a stage. Without the timestamps, the team would be guessing.

Quick check
Ten turns were timed, in milliseconds, and one was very slow. What does this print: the typical turn (the middle of the sorted list), and the average?
timings = sorted([900, 700, 800, 3100, 750, 820, 780, 760, 810, 790])
print(timings[len(timings) // 2], sum(timings) / len(timings))
  1. 1021 800.0
  2. 800 1021.0
  3. 800 800.0
  4. 3100 1021.0
Show the answer

800 1021.0

Sorted, the sixth value is 800: a typical turn. The average is 1,021, pulled up by the single turn of 3,100 ms. Neither number describes that turn. The typical value ignores it and the average smears it across all ten, which is why a slow-turn figure, such as the slowest one in twenty, is reported beside them.

Worked example

The latency of many turns, with the typical and the slow marked

500 simulated turns with a few slow ones in the tail. The average sits between the typical turn and the slow ones, and describes neither.

import random
import matplotlib.pyplot as plt

rng = random.Random(0)
turns = sorted(rng.lognormvariate(6.9, 0.35) for _ in range(500))
p50, p95 = turns[250], turns[475]
print("typical (p50):", round(p50), "ms")
print("slow one in twenty (p95):", round(p95), "ms")
print("average:", round(sum(turns) / len(turns)), "ms")
fig, ax = plt.subplots(figsize=(6.5, 3))
ax.hist(turns, bins=30)
ax.axvline(p50, color="green", label="p50")
ax.axvline(p95, color="red", label="p95")
ax.set_xlabel("silence before the agent speaks (ms)")
ax.set_ylabel("turns")
ax.set_title("500 turns")
ax.legend()
In the lesson you can change this and run it.

The typical turn is 1,027 ms, the slow one in twenty is 1,779 ms, and the average of 1,064 ms lies between them without describing either. A team that reports the average hides the 1 in 20 callers who wait nearly 1.8 seconds. Report p50 and p95, and chart the whole spread.

Worked example

Streaming sentences to the speech engine

A stub for the model's token stream, and the splitter that sends each finished sentence on at once instead of waiting for the whole reply.

def sentences(tokens):
    buf = ""
    for tok in tokens:
        buf += tok
        if tok.endswith((".", "?", "!")):
            yield buf.strip()
            buf = ""
    if buf.strip():
        yield buf.strip()

words = ["Sure", ",", " I", " can", " move", " it", " to", " Thursday", ".",
         " Is", " ten", " ok", "?"]
for sentence in sentences(iter(words)):
    print("send to the speech engine:", sentence)
In the lesson you can change this and run it.

The first sentence leaves after 9 tokens, long before the second is written, so the voice can start while the model is still working. This is the whole idea of streaming between stages: each stage hands on small pieces as soon as they are complete.

Where this shows up in an engineer's week

On the job: a voice product lives or dies on its tail. Log every turn's timestamps, chart p50 and p95 per day, and alert when p95 moves, not the average. When someone says the agent feels slow, split the silence by stage from the log before you touch the model. And test interruptions on purpose: play audio over the agent's reply and check that it stops and that the history holds only the words that were heard.

Goes deeper in

In your own words
You are asked why a voice agent "feels slow". What do you measure, and in what order do you look for the cause?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 7 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 14 review cards for spaced repetition

Try a quiz question

What does streaming every stage of a voice agent change?

  1. The time until the first sound, not the total work
  2. The total time the model takes to write its reply
  3. The accuracy of the speech recognizer
  4. The number of tokens the reply costs

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 5 · about 36 hours of study

Transformers

Attention, GPT from scratch and pretraining 47 lessons 1 free
  1. Language modeling FreePredict the next token, and score it with cross-entropy.30 min
  2. Tokenization and BPEHow tokenizers are built from data.40 min
  3. A bigram modelThe simplest language model, built from counts.35 min
  4. Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
  5. Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
  6. Neural language modelsWhy counting fails, and what embeddings fix.35 min
  7. Attention: the intuitionEach token decides which other tokens matter to it.30 min
  8. Queries, keys and valuesThe three projections behind attention.35 min
  9. Scaled dot-product attentionThe attention formula, line by line.35 min
  10. Causal maskingStop tokens from seeing the future.25 min
  11. Multi-head attentionSeveral attention patterns in parallel.40 min
  12. Positional encodingsTell the model where each token sits.30 min
  13. The transformer blockAttention, MLP, residuals and norms together.40 min
  14. The GPT architectureA stack of decoder blocks that predicts the next token.40 min
  15. Build a tiny GPTWrite and train a small GPT from scratch.45 min
  16. From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
  17. Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
  18. Encoders and embedding modelsBERT-style models and where they fit.40 min
  19. Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
  20. Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
  21. Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
  22. The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
  23. The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
  24. Pretraining at scaleData, compute and what models learn from the internet.40 min
  25. Scaling lawsHow loss falls predictably with size, data and compute.40 min
  26. Inference and the KV cacheWhy generation is fast after the first token.40 min
  27. The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
  28. Long contextEfficient attention and million-token windows.35 min
  29. Mixture of expertsRoute each token to a few specialist sub-networks.35 min
  30. Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
  31. Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
  32. Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
  33. The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
  34. Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
  35. Multimodal transformersHow models see images and hear audio.35 min
  36. Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
  37. Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
  38. Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
  39. Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
  40. Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
  41. Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
  42. Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
  43. Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
  44. Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
  45. Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
  46. Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
  47. The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write