The voice agent loop and its latency budget
One turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.
What it covers8
Why it matters
A voice agent is a chat agent with ears and a mouth: recognition in front of the model, synthesis after it. Wiring the three together takes an afternoon. Making the result pleasant to talk to is a different job, and almost all of it is about time. People answer each other within a fraction of a second, notice a pause of one second, and give up on a system that leaves three. They also interrupt, and stop to think in the middle of a sentence. This lesson is about the loop that copes with all of that, and the arithmetic of where its milliseconds go.
What you'll be able to do
- Describe one turn of a voice agent: listening, deciding the person has finished, thinking, speaking
- Add up a latency budget, and show why streaming every stage changes it
- Explain the trade in deciding when someone has stopped talking, and how to improve on a fixed wait
- Handle an interruption: stop speaking, and record only what was heard
- Read a turn's log, find the slowest stage, and judge latency by its slow turns as well as its typical ones
Show the answer
Synthesizing the whole reply before playing it
Sixteen seconds of speech made four times faster than real time still takes four seconds to make. If nothing is played until all of it exists, those four seconds are silence, far more than recognition or the model's first token. The cure is not a faster synthesizer. It is to start playing the first sentence while the rest is still being made.
The loop: one turn, and where the time goes
This rests on Recognition and synthesis as a pipeline, the stage that turns audio into words, and on Streaming responses, which sends a model's reply token by token.
One turn
One turn of a voice agent has four parts.
- Listen. Audio streams in from the microphone.
- Decide the person has finished. Called endpointing: the agent waits for a stretch of silence.
- Think. The words go to the model, as in any chat.
- Speak. The reply is synthesized and played.
What the person experiences is one number: the silence between the end of their sentence and the start of the agent's voice. The cell adds it up for two designs. The stage times are assumptions for an ordinary setup, chosen to compare the designs.
# One turn of a voice agent, in milliseconds. The numbers are assumptions for a typical # setup, chosen to compare two designs. TOKENS = 52 # the reply as the model writes it: about 40 words TOKENS_PER_SECOND = 50 FIRST_SENTENCE_TOKENS = 11 # "Sure, I can move that to Thursday." or the like # how long the agent waits, after the last sound, before # deciding the person has finished wait_for_silence = 700 one_at_a_time = [ ("wait for silence", wait_for_silence), ("recognize the whole recording", 400), ("model: first token", 400), ("model: the rest of the reply", round(1000 * TOKENS / TOKENS_PER_SECOND)), # 16 seconds of speech, made four times faster than it plays ("synthesize the whole reply", 4000), ] streaming = [ ("wait for silence", wait_for_silence), ("finish recognizing (it ran while they spoke)", 100), ("model: first token", 400), ("model: the first sentence", round(1000 * FIRST_SENTENCE_TOKENS / TOKENS_PER_SECOND)), ("synthesize the first sentence", 150), ] for name, stages in (("each stage waits for the one before to finish", one_at_a_time), ("every stage streams", streaming)): total = sum(ms for stage, ms in stages) print(name + ": silence before the agent starts to speak =", total, "ms") for stage, ms in stages: print(" ", stage.ljust(46), str(ms).rjust(5), "ms", str(round(100 * ms / total)).rjust(3) + "%")
With each stage waiting for the last to finish, the silence is 6,540 ms, and 61% of it is synthesis: the whole reply is made before any of it plays.
Streaming changes what each stage waits for. Recognition runs while the person talks, so only a little is left to finish at the end. The model's tokens are passed on as they appear. The synthesizer starts on the first sentence as soon as it is complete, and playback begins while later sentences are still being written. The silence drops to 1,570 ms.
The model is not one millisecond faster. The total work is the same. Only the time to the first sound has changed, and that is the number people feel.
Look at what is now the largest item: 45% of the streaming budget is spent waiting to be sure the person has stopped. The next part is about that wait.
Show the answer
The reader can start reading long before the reply is finished
Streaming doesn't speed the model up. It lets the person start reading as soon as the first words exist, so the wait they feel is the time to the first token and not the time to the last. A voice agent applies the same idea at every stage, and the stakes are higher: unread text on a screen is fine, and silence in a conversation is not.
Write a sentence or two, then compare it with a good answer in the lesson.
In code: endpoints and interruptions
Have they finished?
A chat app knows when you are done: you press Enter. A voice agent has to guess, from silence. And people fall silent in the middle of sentences all the time: "Can you move my appointment to... Thursday."
So endpointing is a wait: if nobody has spoken for this long, the turn is over. Too short, and the agent talks over people who were only thinking. Too long, and every turn starts with dead air. The cell simulates 2,000 things people say, each with pauses inside it, and tries five waits. Then it tries a smarter rule.
import random def utterance(rng): """The pauses inside one thing a person says, in ms. People stop to think in the middle of sentences.""" return [rng.lognormvariate(5.3, 0.7) for i in range(rng.randint(1, 5))] rng = random.Random(0) talk = [utterance(rng) for i in range(2000)] longest = sorted(max(p) for p in talk) print("longest pause inside an utterance: half are under", round(longest[1000]), "ms, and one in ten is over", round(longest[1800]), "ms") print() print("wait this long | cut the person off | silence added to every turn") for wait in (200, 400, 700, 1000, 1500): cut = sum(max(p) >= wait for p in talk) / len(talk) print(str(wait).rjust(11), "ms |", f"{100 * cut:5.1f}% of turns".rjust(18), "|", wait, "ms") # A smarter rule: read the words so far. If they sound finished, wait a little. If # they don't, wait longer. rng = random.Random(1) cut = waited = 0 for pauses in talk: # at each pause inside the utterance the sentence is NOT finished; the reader gets # that right 9 times in 10 cut += any(p >= (1500 if rng.random() < 0.9 else 300) for p in pauses) # at the real end the sentence IS finished; the reader gets that right 9 times in 10 waited += 300 if rng.random() < 0.9 else 1500 print() print("read the words too |", f"{100 * cut / len(talk):5.1f}% of turns".rjust(18), "|", round(waited / len(talk)), "ms on average")
There is no good fixed wait. At 400 ms the agent cuts into 38.7% of turns. At 1,000 ms it interrupts only 2.8%, and adds a full second of silence to every turn, which is most of a comfortable budget gone before the model has seen a word.
The last line uses something silence doesn't have: the words. "Can you move my appointment to" is plainly unfinished, and "to Thursday" plainly is. A small model that reads the transcript so far, and waits 300 ms if it sounds complete and 1,500 ms if it doesn't, cuts off 9.2% of turns while adding 417 ms on average. That beats the fixed wait of 700 ms on both counts, with a reader that is wrong one time in ten.
This is why voice systems have a turn detector that looks at more than loudness: the words, and in good ones the melody of the voice, which falls at the end of a statement. A separate, simpler piece, voice activity detection, only answers "is anyone speaking right now?".
Show the answer
Every turn, including quick ones, starts 800 ms later
The wait is paid on every turn, since the agent can't know in advance which silences are endings. A longer wait fixes the interruptions and makes the whole conversation feel sluggish, even for "yes" and "no". The better fix decides from the words whether a short or a long wait is right for this particular pause.
Silence before the first sound
Write first_audio_ms(wait_ms, finish_asr_ms, first_token_ms, tokens_per_second, first_sentence_tokens, tts_ms): the silence, in milliseconds, between the person finishing and the agent's first audio in a fully streaming design. It is the wait for silence, the time to finish recognizing, the model's time to its first token, the time to write the first sentence at tokens_per_second, and the synthesizer's time for that sentence. Round the result to a whole number.
def first_audio_ms(wait_ms, finish_asr_ms, first_token_ms, tokens_per_second, first_sentence_tokens, tts_ms): return 0 print(first_audio_ms(700, 100, 400, 50, 11, 150))
When does the turn end?
A voice activity detector gives one value for each 20 ms frame: True if someone is speaking. Write endpoint(frames, quiet_needed): the index of the frame at which the turn is declared over. That is the first frame that completes a run of quiet_needed silent frames in a row after speech has started. Silence before the person first speaks doesn't count. Return None if the turn never ends.
def endpoint(frames, quiet_needed): return None frames = [False, False, True, True, True, False, False, True, True, False, False, False, False] print(endpoint(frames, 3))
Being interrupted
People talk over each other, and a voice agent that keeps reading out its answer while you say "no, wait" is unusable. Handling an interruption, called barge-in, has three parts.
- Notice. Keep listening while speaking. That needs echo cancellation, or the agent hears its own voice from the loudspeaker and interrupts itself.
- Stop. Cut the audio at once, and cancel the reply still being generated and synthesized.
- Remember correctly. The conversation history must hold what the person heard, which is less than what the model wrote.
The cell walks through one interrupted turn as a small state machine.
reply = ("Your order shipped on Monday and should arrive Thursday. I can also send the " "tracking link by email, " "or change the delivery address if you will be " "away. Would you like me to " "do either of those?").split() WORDS_PER_SECOND = 2.5 state, log = "idle", [] def move(ms, new, why): global state log.append((ms, state, new, why)) state = new move(0, "listening", "the person starts to speak") move(2100, "thinking", "end of their turn detected") move(3200, "speaking", "first audio of the reply") speech_started = 3200 barge_in_heard = 3200 + 4800 # 4.8 s into the reply, the person starts talking over it stopped = barge_in_heard + 200 # the agent needs 200 ms to notice and stop the audio move(stopped, "listening", "the person interrupted") for ms, before, after, why in log: print(str(ms).rjust(5), "ms |", before.ljust(9), "->", after.ljust(9), "|", why) heard = int((stopped - speech_started) / 1000 * WORDS_PER_SECOND) print() print("the reply had", len(reply), "words. The person heard", heard, "of them:") print(" heard :", " ".join(reply[:heard]) + " ...") print(" not heard:", "... " + " ".join(reply[heard:])) print("what goes in the conversation history: the", heard, "words heard, marked as interrupted. Not all", len(reply))
The agent moved from listening to thinking to speaking, and back to listening when the person spoke. The interruption came 4.8 seconds into the reply, and the audio stopped 200 ms after that.
The model wrote 37 words. The person heard 12. If the history records all 37, then on the next turn the model believes it offered to send a tracking link and to change the address. The person heard neither offer. When they say "yes please", the model will act on an offer that, for them, was never made.
So the history is cut to the words that were actually played, and marked as interrupted. Getting this right needs the synthesizer to report how far playback got. It is the bug that most separates a demo from a product.
started, stopped, words_per_second = 3200, 8200, 2.5 print(int((stopped - started) / 1000 * words_per_second))
Show the answer
12
The audio played for 5,000 ms, which is 5 seconds, and at 2.5 words a second that is 12.5 words: 12 whole ones. Those 12 are what the conversation history should hold for this turn, whatever the model had planned to say.
Write a sentence or two, then compare it with a good answer in the lesson.
In production: the slow turns
Budget for the slow turns
A latency budget written with one number per stage describes a typical turn. People don't judge an agent by its typical turn. They remember the one where it sat silent for three seconds.
The cell runs 5,000 streaming turns in which every stage's time varies, as real ones do. The spread given to each stage is an assumption. The model's first token is given the widest, as is usual when a model is shared and busy.
import random rng = random.Random(0) def turn(): """One streaming turn. Each stage's time varies from turn to turn, and the model's first token varies most.""" return {"wait for silence": 300 + rng.expovariate(1 / 60), "finish recognizing": rng.lognormvariate(4.5, 0.3), "model: first token": rng.lognormvariate(5.8, 0.6), "model: first sentence": rng.lognormvariate(5.3, 0.3), "synthesize it": rng.lognormvariate(5.0, 0.3)} turns = [turn() for i in range(5000)] totals = sorted(sum(t.values()) for t in turns) print("silence before the agent speaks | typical turn:", round(totals[2500]), "ms | 1 turn in 20 is slower than:", round(totals[4750]), "ms | 1 in 100:", round(totals[4950]), "ms") print("turns that keep within 1,000 ms:", f"{100 * sum(t <= 1000 for t in totals) / 5000:.1f}%", "| within 1,500 ms:", f"{100 * sum(t <= 1500 for t in totals) / 5000:.1f}%") slow = [t for t in turns if sum(t.values()) > totals[4750]] print() print("stage".ljust(22), "| typical turn | the slowest 1 turn in 20") for stage in turns[0]: typical = sorted(t[stage] for t in turns)[2500] print(stage.ljust(22), "|", str(round(typical)).rjust(9), "ms |", str(round(sum(t[stage] for t in slow) / len(slow))).rjust(6), "ms")
A typical turn has 1,162 ms of silence. One turn in 20 has more than 1,748 ms, and one in 100 more than 2,161. In a conversation of 20 turns, the person will most likely meet one of those slow turns.
The second table shows where a slow turn's time went. Four of the five stages are about as fast in the slowest turns as in typical ones. One isn't: the model's first token, at 333 ms typically and 1,147 ms in the slow turns. The tail of the whole system is the tail of one stage.
That tells you what to do. Averages are improved by shaving every stage. Tails are improved by attacking the one stage with the spread: a smaller or less loaded model, a shorter prompt, a cached prefix, or a second request raced against the first. Some agents say a short filler, "let me check", the moment a turn ends. It doesn't make anything faster, and it turns silence into speech.
One more design is worth knowing. Speech-to-speech models take audio in and give audio out with one network, with no text in between. They can be faster and can hear tone of voice. They are also harder to inspect, since there is no transcript in the middle to read, test or filter.
Show the answer
A few very slow turns can hide behind a good average
An average of 900 ms can come from most turns at 700 ms and one in twenty at 3 seconds. In a conversation of twenty turns the person will probably hit one of those, and it is the one they remember. The 95th percentile says how bad that turn is: 95% of turns are faster.
Worked example: read one turn's log
Every voice system worth running writes a timestamp at each handoff. Here is one turn's log, and the few lines that turn it into a latency budget. The person asked to move an appointment, and interrupted the reply partway through.
events = [ # one turn of a voice agent, as its log recorded it {"t": 0, "event": "user_speech_start"}, {"t": 1840, "event": "user_speech_end"}, {"t": 2140, "event": "endpoint"}, {"t": 2230, "event": "asr_final", "text": "can you move my appointment to thursday"}, {"t": 2780, "event": "llm_first_token"}, {"t": 3010, "event": "llm_first_sentence", "text": "Sure, I can move it to Thursday."}, {"t": 3170, "event": "tts_first_audio"}, {"t": 5900, "event": "barge_in"}, {"t": 6080, "event": "tts_stopped"}, ] at = {e["event"]: e["t"] for e in events} stages = [("wait for silence", "user_speech_end", "endpoint"), ("finish recognizing", "endpoint", "asr_final"), ("model: first token", "asr_final", "llm_first_token"), ("model: first sentence", "llm_first_token", "llm_first_sentence"), ("synthesize it", "llm_first_sentence", "tts_first_audio")] total = at["tts_first_audio"] - at["user_speech_end"] print("the person stopped at", at["user_speech_end"], "ms and the agent's voice", "started at", at["tts_first_audio"], "ms: a silence of", total, "ms") for name, start, end in stages: ms = at[end] - at[start] print(" ", name.ljust(22), str(ms).rjust(4), "ms", "#" * round(ms / 25)) print("the largest share:", max(stages, key=lambda s: at[s[2]] - at[s[1]])[0]) print("the person interrupted", at["barge_in"] - at["tts_first_audio"], "ms into the", "reply, and the audio stopped", at["tts_stopped"] - at["barge_in"], "ms later")
The person stopped speaking at 1,840 ms and heard the agent at 3,170: a silence of 1,330 ms. The breakdown says where it went. The wait for silence was 300 ms, short, so this system is probably using the words to decide. Finishing recognition took 90 ms, which means it was streaming all along. The model's first token took 550 ms, the largest share and well above a typical 333, so this was one of the slower turns and the model is the stage to look at. The first sentence took another 230 ms and its synthesis 160. Then the interruption: 2,730 ms into the reply, with the audio stopping 180 ms later. That last number is a budget of its own. Much over a quarter of a second, and the agent seems not to be listening. Five subtractions turned a vague "it felt slow" into the name of a stage. Without the timestamps, the team would be guessing.
timings = sorted([900, 700, 800, 3100, 750, 820, 780, 760, 810, 790]) print(timings[len(timings) // 2], sum(timings) / len(timings))
Show the answer
800 1021.0
Sorted, the sixth value is 800: a typical turn. The average is 1,021, pulled up by the single turn of 3,100 ms. Neither number describes that turn. The typical value ignores it and the average smears it across all ten, which is why a slow-turn figure, such as the slowest one in twenty, is reported beside them.
The latency of many turns, with the typical and the slow marked
500 simulated turns with a few slow ones in the tail. The average sits between the typical turn and the slow ones, and describes neither.
import random import matplotlib.pyplot as plt rng = random.Random(0) turns = sorted(rng.lognormvariate(6.9, 0.35) for _ in range(500)) p50, p95 = turns[250], turns[475] print("typical (p50):", round(p50), "ms") print("slow one in twenty (p95):", round(p95), "ms") print("average:", round(sum(turns) / len(turns)), "ms") fig, ax = plt.subplots(figsize=(6.5, 3)) ax.hist(turns, bins=30) ax.axvline(p50, color="green", label="p50") ax.axvline(p95, color="red", label="p95") ax.set_xlabel("silence before the agent speaks (ms)") ax.set_ylabel("turns") ax.set_title("500 turns") ax.legend()
The typical turn is 1,027 ms, the slow one in twenty is 1,779 ms, and the average of 1,064 ms lies between them without describing either. A team that reports the average hides the 1 in 20 callers who wait nearly 1.8 seconds. Report p50 and p95, and chart the whole spread.
Streaming sentences to the speech engine
A stub for the model's token stream, and the splitter that sends each finished sentence on at once instead of waiting for the whole reply.
def sentences(tokens): buf = "" for tok in tokens: buf += tok if tok.endswith((".", "?", "!")): yield buf.strip() buf = "" if buf.strip(): yield buf.strip() words = ["Sure", ",", " I", " can", " move", " it", " to", " Thursday", ".", " Is", " ten", " ok", "?"] for sentence in sentences(iter(words)): print("send to the speech engine:", sentence)
The first sentence leaves after 9 tokens, long before the second is written, so the voice can start while the model is still working. This is the whole idea of streaming between stages: each stage hands on small pieces as soon as they are complete.
Where this shows up in an engineer's week
On the job: a voice product lives or dies on its tail. Log every turn's timestamps, chart p50 and p95 per day, and alert when p95 moves, not the average. When someone says the agent feels slow, split the silence by stage from the log before you touch the model. And test interruptions on purpose: play audio over the agent's reply and check that it stops and that the history holds only the words that were heard.
Goes deeper in
- Recognition and synthesis as a pipeline: the recognition stage and what it costs per second of audio.
- Streaming responses: sending a model's reply as it is written.
- Rate limits, retries and idempotency: timeouts and retries, which decide what a slow turn becomes.
Write a sentence or two, then compare it with a good answer in the lesson.
Practice what you just read
- 7 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 14 review cards for spaced repetition
Try a quiz question
What does streaming every stage of a voice agent change?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 5 · about 36 hours of study
Transformers
Attention, GPT from scratch and pretraining
47 lessons 1 free
- Language modeling FreePredict the next token, and score it with cross-entropy.30 min
- Tokenization and BPEHow tokenizers are built from data.40 min
- A bigram modelThe simplest language model, built from counts.35 min
- Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
- Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
- Neural language modelsWhy counting fails, and what embeddings fix.35 min
- Attention: the intuitionEach token decides which other tokens matter to it.30 min
- Queries, keys and valuesThe three projections behind attention.35 min
- Scaled dot-product attentionThe attention formula, line by line.35 min
- Causal maskingStop tokens from seeing the future.25 min
- Multi-head attentionSeveral attention patterns in parallel.40 min
- Positional encodingsTell the model where each token sits.30 min
- The transformer blockAttention, MLP, residuals and norms together.40 min
- The GPT architectureA stack of decoder blocks that predicts the next token.40 min
- Build a tiny GPTWrite and train a small GPT from scratch.45 min
- From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
- Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
- Encoders and embedding modelsBERT-style models and where they fit.40 min
- Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
- Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
- Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
- The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
- The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
- Pretraining at scaleData, compute and what models learn from the internet.40 min
- Scaling lawsHow loss falls predictably with size, data and compute.40 min
- Inference and the KV cacheWhy generation is fast after the first token.40 min
- The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
- Long contextEfficient attention and million-token windows.35 min
- Mixture of expertsRoute each token to a few specialist sub-networks.35 min
- Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
- Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
- Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
- The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
- Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
- Multimodal transformersHow models see images and hear audio.35 min
- Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
- Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
- Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
- Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
- Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
- Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
- Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
- Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
- Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
- Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
- Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
- The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write