Recognition and synthesis as a pipeline
A whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.
What it covers6
Why it matters
Speech recognition looks like magic until it is taken apart, and then it looks like plumbing: sound becomes features, features become a guess at the sound in each instant, the guesses are squeezed into a sequence, and the sequence is matched against the words that exist. Each stage has its own way of failing, and knowing which stage failed is most of debugging a voice product. This lesson builds the whole pipeline for a made-up language of six sounds and twelve words: small enough to read every line, and real enough to break in the ways real recognizers do.
What you'll be able to do
- Name the stages of a speech recognizer, and what passes between them
- Explain why audio frames and text don't line up, and how collapsing repeats and blanks solves it
- Measure recognition with word error rate, and count its three kinds of error
- Show what a dictionary or language model adds when the sound alone is unclear
- Describe speech synthesis as the same pipeline run in the other direction
Show the answer
About 25
Half a second is 500 ms, and 500 ÷ 20 is 25 frames. So about 25 frames have to become 4 letters, and nothing says which frames belong to which letter, or how long each sound was held. That mismatch between a long input and a short output, with no alignment given, is the central problem of speech recognition.
The idea: two sequences that don't line up
This rests on Sound as numbers: waveforms and spectrograms for frames and spectra, and on Language modeling for scoring which word comes next.
Two sequences that don't line up
A recognizer has three jobs, done in order.
- Features. Cut the sound into frames and describe each one, as in Sound as numbers: waveforms and spectrograms.
- An acoustic model. For each frame, say which sound it most likely is.
- A decoder. Turn the long string of per-frame guesses into words.
Real languages have dozens of sounds, and the acoustic model is a neural network. To keep every part readable, this lesson uses a made-up language with six sounds, each a pair of tones, and an acoustic model that just measures the energy at each sound's two tones. The function speak makes the audio.
Setup (runs before every cell) · 66 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import numpy as np RATE, FRAME = 8000, 160 # 8,000 samples a second, frames of 20 ms # A made-up language with six sounds. Each sound is two tones at once. SOUNDS = {"a": (700, 1200), "e": (500, 1900), "o": (450, 900), "k": (1400, 2600), "s": (2200, 3300), "m": (300, 1000)} # The dictionary: how each word is written, and the sounds it is made of. LEXICON = {"sam": "sam", "some": "som", "mask": "mask", "seem": "seem", "make": "mek", "oak": "ok", "ask": "ask", "keks": "keks", "moose": "moos", "mess": "mes", "sake": "sek", "coma": "koma"} def speak(sounds, rng=None, ms=80, gap=40): """Synthesis: each sound becomes about 80 ms of its two tones, with a short silence after it.""" parts = [] for s in sounds: # people don't hold every sound equally long length = ms if rng is None else ms * rng.uniform(0.7, 1.3) t = np.arange(int(RATE * length / 1000)) / RATE f1, f2 = SOUNDS[s] parts += [0.5 * np.sin(2 * np.pi * f1 * t) + 0.5 * np.sin(2 * np.pi * f2 * t), np.zeros(int(RATE * gap / 1000))] return np.concatenate(parts) def spectrum(x): frames = np.array([x[i : i + FRAME] for i in range(0, len(x) - FRAME + 1, FRAME)]) # one row per frame, frequencies 50 Hz apart return np.abs(np.fft.rfft(frames * np.hanning(FRAME), axis=1)) def label_frames(spec): """The acoustic model: one label per frame. The sound whose two tones hold the most energy, or '-' if the frame is quiet or unclear.""" scores = np.array([[frame[f1 // 50] + frame[f2 // 50] for f1, f2 in SOUNDS.values()] for frame in spec]) best = scores.max(axis=1) loud = best > 0.5 * np.median(np.sort(best)[-max(3, len(best) // 4):]) names, out = list(SOUNDS), [] for row, ok in zip(scores, loud): ranked = np.sort(row) out.append( names[int(row.argmax())] if ok and ranked[-1] > 1.25 * ranked[-2] else "-") return "".join(out) def collapse(labels): """Merge repeats, then drop the blanks: 'mm--aa-ss' becomes 'mas'.""" out, previous = [], None for label in labels: if label != previous and label != "-": out.append(label) previous = label return "".join(out) def distance(a, b): """Edit distance: the fewest insertions, deletions and substitutions that turn a into b.""" row = list(range(len(b) + 1)) for i, x in enumerate(a, 1): new = [i] for j, y in enumerate(b, 1): new.append(min(row[j] + 1, new[j - 1] + 1, row[j - 1] + (x != y))) row = new return row[-1] def nearest_word(sounds): """The dictionary word whose sounds are closest to what was heard.""" return min(sorted(LEXICON), key=lambda word: distance(sounds, LEXICON[word]))
for word in ("mask", "some", "coma"): sounds = LEXICON[word] x = speak(sounds) labels = label_frames(spectrum(x)) print(word.ljust(5), "| letters:", len(word), "| sounds:", sounds.ljust(4), "| samples:", len(x), "| frames:", len(labels), "|", labels)
Look at the lengths. "mask" is 4 letters, 4 sounds, 3,840 samples and 24 frames. The frame labels read mmmm--aaaa--ssss--kkkk--: each sound lasts about four frames, and a dash marks a frame that was silent or unclear.
Two things here are true of real speech as well.
Spelling isn't sound. "some" is four letters and three sounds, written som. The table that maps each word to its sounds is a pronunciation dictionary, or lexicon, and the units of sound are called phonemes.
Nothing marks the boundaries. The audio doesn't say where one sound ends and the next begins, and a slow speaker would give forty frames where a fast one gives sixteen. Whatever turns frames into text has to cope with any stretching.
Show the answer
How probable the sequence is
A language model assigns a probability to a sequence, by predicting each word from the ones before. A recognizer uses that to choose between guesses that sound alike: "recognize speech" is far more probable than "wreck a nice beach", though the two are nearly the same sounds. Later in this lesson a plain dictionary plays that part.
Press Next. The lit arrow is the step described; the label under it is the output of that step for the word "mask".
Write a sentence or two, then compare it with a good answer in the lesson.
In PyTorch: frames, sounds and a dictionary
From frames to sounds
The acoustic model gives one label per frame. Turning mmmm--aaaa--ssss--kkkk-- into mask takes one rule: merge repeated labels, then drop the blanks.
Uses the setup above, 66 lines.
for sounds in ("mask", "moos"): labels = label_frames(spectrum(speak(sounds))) print(sounds, "| frame labels:", labels, "| collapsed:", collapse(labels)) labels = label_frames(spectrum(speak("moos"))) no_blanks = labels.replace("-", "") print() print("moos with the blanks thrown away first:", no_blanks, "| collapsed:", collapse(no_blanks)) rng = np.random.default_rng(3) for trial in range(3): labels = label_frames(spectrum(speak("koma", rng))) print("koma, said a little differently:", labels.ljust(30), "|", len(labels), "frames | collapsed:", collapse(labels))
The rule handles any speed. The word "koma" is said three times with each sound held for a different length, giving 23, 21 and 23 frames with the labels bunched differently, and all three collapse to koma.
The blank is doing real work. "moos" has the same sound twice in a row. With blanks between the two runs of o, merging keeps them apart, and the result is moos. Throw the blanks away first and the two runs join into one: mos, a different word.
This is the idea behind CTC, the method that let neural networks be trained for speech without anyone marking where each sound starts. The network outputs a label or a blank for every frame. Many different frame strings collapse to the same text, and training counts them all as correct. The network is free to decide for itself how long each sound lasts.
Taking the single most likely label in each frame and collapsing is called greedy decoding. It is the simplest decoder, and the baseline every better one is compared with.
def collapse(labels): out, previous = [], None for label in labels: if label != previous and label != "-": out.append(label) previous = label return "".join(out) print(collapse("hh-e-ll-ll-oo"), collapse("hhelllloo"), collapse("--a--"))
Show the answer
hello helo a
In the first string a blank separates the two runs of l, so both survive: hello. In the second there is no blank, the four l's are one run, and they merge into one: helo. Leading and trailing blanks simply vanish. A double letter can only be written if a blank falls between its two halves.
Decode an acoustic model's output
An acoustic model returns, for each frame, one score per label. Write greedy_decode(scores, labels). labels is a string such as "-aeo", whose first character is the blank. For each frame take the label with the highest score (the earliest, if several tie). Then merge repeats and drop the blanks, and return the text.
Uses the setup above, 66 lines.
def greedy_decode(scores, labels): return "" frames = [[0.1, 0.8, 0.1], [0.2, 0.7, 0.1], [0.9, 0.05, 0.05], [0.1, 0.1, 0.8]] print(greedy_decode(frames, "-ab"))
Noise, and the help of a dictionary
On clean audio the toy recognizer is perfect. The cell adds noise, louder and louder. The signal-to-noise ratio says how much louder the speech is than the noise, in decibels: 0 dB means equally loud, and negative means the noise is louder.
Each word is scored three ways: the share of sounds wrong, the share of words wrong if the sounds are taken exactly as heard, and the share wrong if the decoder picks the nearest word in the dictionary.
Uses the setup above, 66 lines.
def trial(snr_db, n, seed): """n words spoken with noise added. snr_db: how much louder the speech is than the noise, in decibels.""" rng = np.random.default_rng(seed) words = sorted(LEXICON) wrong_sounds = sounds = as_heard = with_dictionary = 0 for i in range(n): word = words[rng.integers(len(words))] x = speak(LEXICON[word], rng) noise = rng.normal(0, np.sqrt((x ** 2).mean() / 10 ** (snr_db / 10)), len(x)) heard = collapse(label_frames(spectrum(x + noise))) wrong_sounds += distance(heard, LEXICON[word]) sounds += len(LEXICON[word]) as_heard += heard != LEXICON[word] with_dictionary += nearest_word(heard) != word return wrong_sounds / sounds, as_heard / n, with_dictionary / n print("speech over noise | sounds wrong | words wrong, " "taken as heard | words wrong, nearest word in the dictionary") for snr_db in (10, 0, -5, -8, -10, -12): sound_errors, as_heard, with_dictionary = trial(snr_db, 300, 1) print((str(snr_db) + " dB").rjust(17), "|", f"{100 * sound_errors:5.1f}%".rjust(12), "|", f"{100 * as_heard:5.1f}%".rjust(27), "|", f"{100 * with_dictionary:5.1f}%")
Down to 0 dB nothing goes wrong. Then it falls apart quickly. At −10 dB three quarters of the sounds are wrong, and taken as heard, 90.0% of the words are wrong: one bad sound spoils a whole word.
The last column is the interesting one. At the same −10 dB, picking the nearest dictionary word gets all but 13.7% right. The acoustic model heard kma, and only one word, koma, is one edit away. The dictionary knows which strings of sounds are words, and that knowledge repairs most single mistakes.
Real recognizers use the same help in a stronger form. A language model knows which sequences of words are likely, so it can repair a word that was a perfectly good word and still the wrong one. Modern speech models build this in: the part that writes the text is a language model, conditioned on the sound.
At −12 dB the first column passes 100%. That isn't a mistake. Errors are counted as edits, and when noise makes the recognizer hear sounds that weren't there, there can be more edits than there were sounds.
The spectrum of the word "mask" as a picture, with time across and frequency up. Each sound is two bright horizontal bands, its two tones, and the frame labels read off which band pair is brightest.
Uses the setup above, 66 lines.
import matplotlib.pyplot as plt spec = spectrum(speak("mask")) # (24 frames, 81 frequencies) plt.imshow(spec.T, origin="lower", aspect="auto", extent=[0, len(spec) * 20, 0, 4000]) plt.xlabel("time (ms)") plt.ylabel("frequency (Hz)") print(spec.shape, label_frames(spec))
def distance(a, b): row = list(range(len(b) + 1)) for i, x in enumerate(a, 1): new = [i] for j, y in enumerate(b, 1): new.append(min(row[j] + 1, new[j - 1] + 1, row[j - 1] + (x != y))) row = new return row[-1] reference, heard = "turn the lights off".split(), "turn lights of now".split() print(distance(reference, heard), distance(reference, heard) / len(reference))
Show the answer
3 0.75
Three edits turn one into the other: "the" was deleted, "off" was replaced by "of", and "now" was inserted. The word error rate is edits divided by the length of the reference: 3 ÷ 4 = 0.75. It is divided by the reference's length, not the output's, which is why insertions can push it past 1.
import math for ratio in (4, 0.25): print(round(10 * math.log10(ratio), 1), end=" ")
Show the answer
6.0 -6.0
For power, decibels are 10 times the base-10 logarithm of the ratio. Four times the power is about +6 dB, and a quarter is about −6 dB: the same distance in the other direction. At 0 dB the two are equally loud, and the toy recognizer above was still perfect there.
How loud is the speech against the noise?
Write snr_db(signal, noise). Both are lists of samples. The power of a list is the mean of its squared values. Return 10 times the base-10 logarithm of the signal's power divided by the noise's power. If the noise's power is 0, return float("inf"). If the signal's power is 0 (and the noise's isn't), return float("-inf").
Uses the setup above, 66 lines.
import math def snr_db(signal, noise): return 0.0 print(round(snr_db([2, -2, 2, -2], [1, -1, 1, -1]), 2))
Write a sentence or two, then compare it with a good answer in the lesson.
In real models: synthesis, and the encoder-decoder
Synthesis is the pipeline run backward
Speech synthesis, or text to speech, runs the stages the other way.
- Text to sounds. Look the words up in the lexicon, or predict their sounds, and decide how long each lasts and how the pitch moves.
- Sounds to a spectrogram. A model writes the mel spectrogram frame by frame.
- Spectrogram to waveform. A vocoder fills in the detail a spectrogram doesn't hold.
The function speak in this lesson is a toy version of all three: it looks up two tones for each sound and writes them out as samples. The cell tries it at three speeds, and then sends every word of the language through synthesis and back through recognition.
Uses the setup above, 66 lines.
print("the same word, said at three speeds") for ms in (40, 80, 160): x = speak(LEXICON["coma"], ms=ms) labels = label_frames(spectrum(x)) print(str(ms).rjust(4), "ms a sound |", str(len(x)).rjust(5), "samples |", str(len(labels)).rjust(2), "frames |", collapse(labels), "->", nearest_word(collapse(labels))) print() right = 0 for word in sorted(LEXICON): # text -> sounds -> waveform -> frames -> sounds -> text heard = nearest_word(collapse(label_frames(spectrum(speak(LEXICON[word]))))) right += heard == word print("round trip, synthesis then recognition:", right, "of", len(LEXICON), "words come back as themselves")
At 40, 80 and 160 ms a sound, the audio is 16, 24 and 40 frames long, and all three are recognized as "coma": the collapse rule does not care about speed. And all 12 words survive the round trip. That round trip is a real technique: synthesized speech is used to test recognizers, and recognizers are used to score synthesizers.
Two things make real synthesis hard, and neither shows in a language of tones. One is prosody: the rhythm, stress and melody of a sentence, which change its meaning and which the text doesn't spell out. The other is that natural speech has no silences between sounds. Each sound bends into the next.
Many current systems blur the stages. Some recognizers map log-mel frames straight to text with one network, and some synthesizers and voice models work on learned audio tokens, treating sound as another kind of sequence for a transformer to continue. The stages in this lesson are still what those networks have to do inside, and still the names used when something goes wrong.
Show the answer
The acoustic side: phone audio loses high sounds
The two words differ mainly in their endings, and the hiss of "teen" against "ty" lives at high frequencies. A telephone keeps only up to about 4,000 Hz, so the difference is partly missing before any model runs. That it works on studio audio confirms it. The fix is in the audio and the acoustic model: train on telephone-quality speech.
Worked example: word error rate on sentences
Words in a sentence run together, so the recognizer has one more job: find where each word ends. Here it cuts wherever five frames in a row are quiet. Three sentences are spoken at three noise levels and scored by word error rate: the fewest word insertions, deletions and substitutions that turn the output into the reference, divided by the number of words in the reference.
Uses the setup above, 66 lines.
def say(words, rng): """A sentence: the words one after another, with a pause of 160 ms between them.""" return np.concatenate( [np.concatenate([speak(LEXICON[w], rng), np.zeros(1280)]) for w in words]) def hear(x): """Recognize a sentence: label the frames, cut at long silences, and look each piece up in the dictionary.""" labels = label_frames(spectrum(x)) pieces, current, quiet = [], "", 0 for label in labels + "-----": quiet = quiet + 1 if label == "-" else 0 current += label if quiet == 5: # five quiet frames in a row: a word has ended if collapse(current): pieces.append(nearest_word(collapse(current))) current = "" return pieces def word_error_rate(reference, heard): # the same edit distance, counted in words return distance(reference, heard) / len(reference) rng = np.random.default_rng(4) sentences = [["sam", "make", "some", "keks"], ["ask", "moose", "sake"], ["mess", "seem", "coma", "oak", "mask"]] for snr_db in (0, -10, -12): errors = total = 0 for sentence in sentences: x = say(sentence, rng) heard = hear( x + rng.normal(0, np.sqrt((x ** 2).mean() / 10 ** (snr_db / 10)), len(x))) errors += distance(sentence, heard) total += len(sentence) print(str(snr_db).rjust(4), "dB |", " ".join(sentence).ljust(26), "->", " ".join(heard).ljust(30), "| word error rate", f"{word_error_rate(sentence, heard):.2f}") print(" all three sentences:", errors, "errors in", total, "words: word error rate", f"{errors / total:.2f}")
At 0 dB all 12 words are right. At −10 dB the word error rate is 0.50, and the first sentence shows why it is so much worse than the 13.7% for single words: "sam make some keks" came back as "sam coma keks". Noise filled the pause between two words, the recognizer never saw five quiet frames, and "make some" was decoded as one word. Finding the boundaries is a failure of its own kind, which isolated words never show. At −12 dB the rate is 0.83, and two sentences score 1.00, as many errors as words. Read the outputs and the kinds of error are there: substitutions (a wrong word in place) and deletions (two words merged into one). The third kind, an insertion, is noise heard as an extra word. One number, the word error rate, covers all three, and its weakness is that it treats them as equally bad. Mishearing "the" and mishearing a name or a number cost the same, so teams also track the errors that matter to their product.
Two ways to get from frames to text
This lesson's recognizer labels every frame, then collapses the labels. That is the idea behind CTC (connectionist temporal classification), which trains such a model by adding up every way the frames could have produced the right text. The card below shows the loss.
Most current recognizers, including Whisper, do something different: an encoder-decoder transformer. The encoder reads the log-mel frames. The decoder writes tokens one at a time and uses cross-attention, the same as in a translation model, to look at the part of the audio it is working on. The alignment between sound and text is learned by attention, so there are no blank labels. Synthesis often goes the other way: text to tokens to a spectrogram to a waveform.
On the job: report word error rate on your own audio, never only on a benchmark's. Record 50 real calls or meetings from the setting where the product will run, get human transcripts, and compute the rate with a tool like jiwer. Then look at the errors, not just the number: names, numbers and product terms often fail first, and a custom word list or a language-model rescoring pass can fix those where more training data can't.
Write a sentence or two, then compare it with a good answer in the lesson.
Goes deeper in
- The voice agent loop and its latency budget: the loop of listening, thinking and speaking, and its latency budget.
- Sound as numbers: waveforms and spectrograms: how sound becomes frames and spectrograms.
- Beam search and constrained decoding: beam search, the usual way to turn per-frame scores into text.
Practice what you just read
- 6 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 13 review cards for spaced repetition
Try a quiz question
What problem does the rule "merge repeats, then drop blanks" solve?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 5 · about 36 hours of study
Transformers
Attention, GPT from scratch and pretraining
47 lessons 1 free
- Language modeling FreePredict the next token, and score it with cross-entropy.30 min
- Tokenization and BPEHow tokenizers are built from data.40 min
- A bigram modelThe simplest language model, built from counts.35 min
- Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
- Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
- Neural language modelsWhy counting fails, and what embeddings fix.35 min
- Attention: the intuitionEach token decides which other tokens matter to it.30 min
- Queries, keys and valuesThe three projections behind attention.35 min
- Scaled dot-product attentionThe attention formula, line by line.35 min
- Causal maskingStop tokens from seeing the future.25 min
- Multi-head attentionSeveral attention patterns in parallel.40 min
- Positional encodingsTell the model where each token sits.30 min
- The transformer blockAttention, MLP, residuals and norms together.40 min
- The GPT architectureA stack of decoder blocks that predicts the next token.40 min
- Build a tiny GPTWrite and train a small GPT from scratch.45 min
- From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
- Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
- Encoders and embedding modelsBERT-style models and where they fit.40 min
- Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
- Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
- Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
- The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
- The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
- Pretraining at scaleData, compute and what models learn from the internet.40 min
- Scaling lawsHow loss falls predictably with size, data and compute.40 min
- Inference and the KV cacheWhy generation is fast after the first token.40 min
- The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
- Long contextEfficient attention and million-token windows.35 min
- Mixture of expertsRoute each token to a few specialist sub-networks.35 min
- Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
- Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
- Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
- The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
- Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
- Multimodal transformersHow models see images and hear audio.35 min
- Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
- Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
- Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
- Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
- Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
- Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
- Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
- Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
- Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
- Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
- Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
- The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write