Sound as numbers: waveforms and spectrograms
A recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.
What it covers6
Why it matters
Speech recognition, voice assistants and music models all start in the same place: a microphone turns moving air into a long list of numbers. A neural network could be handed that list directly, and a few are. Most are handed a picture of it instead, a spectrogram, for the same reason a vision model works on pixels in a grid: the structure that matters is easier to see. This lesson builds that picture from a raw waveform in a few lines of NumPy, so that "the audio encoder takes log-mel features" stops being a phrase and becomes an array whose shape you can predict.
What you'll be able to do
- Describe a recording as an array of samples, and work out its size from the sampling rate
- Explain why a sound must be sampled at more than twice its highest frequency, and what goes wrong otherwise
- Turn a waveform into a spectrogram, and read which frequencies were present when
- Choose a window length, knowing what it trades between pitch and timing
- Build the log-mel features a speech model is given, and say why they are shaped that way
Show the answer
The air pressure at one instant
Sound is air pressure rising and falling. A microphone measures that pressure thousands of times a second, and each measurement is one number, called a sample. Loudness and pitch aren't stored anywhere. They are patterns across many samples: how far the numbers swing, and how quickly.
The idea: a sound is a list of numbers
This rests on Waves: height, speed and shift, where a wave has a frequency and a period, and on Sine and cosine. Here the wave is air pressure.
A sound is air pressure wobbling. A microphone reads the pressure at regular moments, and each reading is a sample. The number of readings per second is the sampling rate, measured in hertz (Hz). The whole recording is one array, a waveform.
Setup (runs before every cell) · 30 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import numpy as np rate = 8000 def stft(x, size, hop): """Cut x into overlapping frames of size samples, hop apart, and take each frame's spectrum.""" window = np.hanning(size) # fades each frame in and out frames = np.array( [x[i : i + size] * window for i in range(0, len(x) - size + 1, hop)]) return np.abs(np.fft.rfft(frames, axis=1)) # rows: frames. columns: frequencies def hz_to_mel(f): return 2595 * np.log10(1 + f / 700) def mel_to_hz(m): return 700 * (10 ** (m / 2595) - 1) def mel_filters(count, size, rate): """count overlapping triangles across the spectrum: narrow at low frequencies, wide at high ones.""" edges = mel_to_hz(np.linspace(0, hz_to_mel(rate / 2), count + 2)) bins = np.fft.rfftfreq(size, 1 / rate) bank = np.zeros((count, len(bins))) for m in range(count): low, middle, high = edges[m], edges[m + 1], edges[m + 2] bank[m] = np.clip( np.minimum((bins - low) / (middle - low), (high - bins) / (high - middle)), 0, None) return bank, edges
import numpy as np rate = 8000 # samples per second t = np.arange(0, 0.01, 1 / rate) # 10 ms of moments at which the sound is measured tone = np.sin(2 * np.pi * 440 * t) # 440 Hz: the air pressure swings 440 times a second print("samples in 10 ms:", len(tone), "| first five:", np.round(tone[:5], 3).tolist()) print("one second at 16,000 samples a second:", 16000, "numbers | a minute:", f"{16000 * 60:,}", "| an hour:", f"{16000 * 3600:,}") print() for amplitude in (1.0, 0.5, 0.1, 0.01): x = amplitude * np.sin(2 * np.pi * 440 * np.arange(0, 1, 1 / rate)) rms = np.sqrt((x ** 2).mean()) # root mean square: the average size of the swings print("amplitude", str(amplitude).ljust(4), "| rms", f"{rms:.4f}", "| in decibels:", f"{20 * np.log10(rms):6.1f}")
Ten milliseconds of sound is 80 numbers at this rate. The first five trace the start of a sine wave rising from zero: a pure tone is the wave you met in the math lessons, with time on the horizontal axis.
Speech models usually work at 16,000 samples a second. An hour of that is 57,600,000 numbers, which is why nobody feeds raw audio to a model without shrinking it first.
The second table is about loudness. Halving the amplitude halves the rms, the typical size of a sample. Ears and engineers both measure loudness on a log scale, in decibels: each halving is 6 dB down, and each division by ten is 20 dB down, as the last column shows. A whisper and a shout are far apart in amplitude and only a few tens of decibels apart.
Show the answer
500 cycles a second
Frequency is 1 ÷ period. One cycle every 0.002 seconds is 1 ÷ 0.002 = 500 cycles a second, or 500 Hz. In sound, frequency is heard as pitch: 440 Hz is the note an orchestra tunes to, and speech lives mostly between about 100 and 4,000 Hz.
rate, seconds, hum = 16000, 2.5, 200 print(int(rate * seconds), rate // hum)
Show the answer
40000 80
Samples are rate times seconds: 16,000 × 2.5 = 40,000. A 200 Hz hum repeats 200 times a second, and a second is 16,000 samples, so one cycle is 16,000 ÷ 200 = 80 samples. The lower the pitch, the more samples each cycle takes.
Write a sentence or two, then compare it with a good answer in the lesson.
How often to measure
How many samples a second are enough? The cell records pure tones of different pitches, always at 8,000 samples a second, and then asks which frequency the recording contains.
Uses the setup above, 30 lines.
import numpy as np rate = 8000 def loudest_hz(x, rate): """The frequency with the most energy in x.""" return float(np.fft.rfftfreq(len(x), 1 / rate)[np.abs(np.fft.rfft(x)).argmax()]) for f in (500, 1000, 3000, 3900, 5000, 7000, 7500): # one second of a tone at f Hz, sampled 8,000 times x = np.sin(2 * np.pi * f * np.arange(0, 1, 1 / rate)) print("a tone of", str(f).rjust(4), "Hz, sampled 8,000 times a second, comes out as", str(round(loudest_hz(x, rate))).rjust(4), "Hz")
Up to 3,900 Hz the recording is faithful. Then it goes wrong in a specific way. A tone at 5,000 Hz comes out as 3,000. One at 7,000 comes out as 1,000. The recording isn't noisy or blurred. It contains a clean tone at the wrong pitch.
This is aliasing. With 8,000 samples a second, a 7,000 Hz wave moves so far between one sample and the next that the samples land exactly where a 1,000 Hz wave would have put them. From the numbers alone, the two can't be told apart.
The limit is half the sampling rate, called the Nyquist frequency: 4,000 Hz here. Anything above it folds back down, as far below the limit as it was above. So the rule is: sample at more than twice the highest frequency you want to keep, and filter out everything above half the rate before sampling.
That rule explains the standard rates. Telephones use 8,000 Hz, keeping speech up to 4,000 Hz: understandable, a little dull. Speech models use 16,000. Music uses 44,100 or more, since people hear up to about 20,000 Hz.
What pitch does the recording hold?
Write apparent_hz(f, rate): the frequency a pure tone of f Hz appears to have after being sampled at rate samples a second. Take the remainder of f divided by rate. If that is more than half the rate, the tone has folded: the answer is rate minus the remainder.
Uses the setup above, 30 lines.
def apparent_hz(f, rate): return f for f in (1000, 5000, 7000): print(f, "->", apparent_hz(f, 8000))
Press Next. The shapes are the ones the worked example at the end of part 2 prints.
Write a sentence or two, then compare it with a good answer in the lesson.
In PyTorch: from waveform to spectrogram
Which frequencies, and when
To find the frequencies in a stretch of samples there is a standard tool, the Fourier transform (np.fft.rfft). It takes a list of samples and returns how much of each frequency they contain: a spectrum.
Applied to a whole recording, it has a blind spot.
Uses the setup above, 30 lines.
import numpy as np rate = 8000 t = np.arange(0, 1.0, 1 / rate) # 500 Hz for half a second, then 1,500 Hz x = np.where(t < 0.5, np.sin(2 * np.pi * 500 * t), np.sin(2 * np.pi * 1500 * t)) whole = np.abs(np.fft.rfft(x)) strongest = np.fft.rfftfreq(len(x), 1 / rate)[np.argsort(whole)[-2:]] print("spectrum of the whole second: strongest at", sorted(np.round(strongest).astype(int).tolist()), "Hz. When? It can't say.") S = stft(x, 256, 128) print("spectrogram:", S.shape[0], "frames by", S.shape[1], "frequencies | a frame every", 1000 * 128 / rate, "ms | frequencies", rate / 256, "Hz apart") for frame in (5, 20, 40, 55): print(" frame", str(frame).rjust(2), "at", f"{frame * 128 / rate:.2f}", "s | loudest:", round(float(S[frame].argmax() * rate / 256)), "Hz")
The sound is 500 Hz for half a second and then 1,500 Hz. The spectrum of the whole second finds both frequencies and has no idea of the order. For speech, order is everything.
The fix is to cut the recording into short overlapping frames and take the spectrum of each. Stack the results and you have a spectrogram: a grid with time along one side and frequency along the other. Here it has 61 frames and 129 frequencies, and it reads the sound correctly: 500 Hz early, 1,500 Hz late.
How long should a frame be? There is no free choice.
Uses the setup above, 30 lines.
import numpy as np rate = 8000 t = np.arange(0, 1.0, 1 / rate) # two tones only 60 Hz apart, for the whole second x = np.sin(2 * np.pi * 1000 * t) + np.sin(2 * np.pi * 1060 * t) x[4000:4008] += 4.0 # and a click 1 ms long, at 0.5 s for size in (64, 256, 1024): hop = size // 4 S = stft(x, size, hop) bins = np.fft.rfftfreq(size, 1 / rate) # the average spectrum, well before the click average = S[: len(S) // 3].mean(axis=0) band = np.where((bins >= 800) & (bins <= 1300))[0] tones = sum(1 for i in band if average[i] > average[i - 1] and average[i] >= average[i + 1] and average[i] > 0.5 * average[band].max()) high = S[:, bins > 2000].sum(axis=1) # only the click reaches up here touched = int((high > 0.2 * high.max()).sum()) print("window of", str(size).rjust(4), "samples =", str(1000 * size // rate).rjust(3), "ms | frequencies", str(round(rate / size, 1)).rjust(5), "Hz apart | tones it can tell apart:", tones, "| the 1 ms click is spread over", str(round(1000 * (touched * hop + size - hop) / rate)).rjust(3), "ms")
This sound has two tones only 60 Hz apart, and a click 1 ms long. An 8 ms window pins the click down to 12 ms and sees the two tones as one. A 128 ms window separates the tones, with frequencies 7.8 Hz apart, and smears the click over 192 ms.
A short frame knows when and is vague about which pitch. A long frame is the reverse. It isn't a flaw in the method: a pitch takes time to exist, so a short stretch of sound doesn't have a precise one. Speech work settles on windows of 20 to 25 ms, moved along 10 ms at a time.
samples, size, hop, rate = 8000, 400, 160, 8000 print((samples - size) // hop + 1, rate / size)
Show the answer
48 20.0
The first frame starts at sample 0, and each next one 160 later, as long as a whole frame of 400 still fits: (8,000 − 400) // 160 + 1 = 48 frames. A frame of 400 samples at 8,000 a second lasts 50 ms, and its spectrum has frequencies 8,000 ÷ 400 = 20 Hz apart. Longer frames give finer frequencies.
How many frames?
Write frame_count(samples, size, hop): how many whole frames of size samples fit in a recording of samples samples, when each frame starts hop samples after the last. A recording shorter than one frame has 0.
Uses the setup above, 30 lines.
def frame_count(samples, size, hop): return 0 print(frame_count(8000, 256, 128), frame_count(48000, 400, 160))
Write a sentence or two, then compare it with a good answer in the lesson.
The spectrogram as a picture
Printing frames hides what a spectrogram is: a picture. The cell draws the waveform above its spectrogram for two window lengths. The sound is 500 Hz, then 1,500 Hz. With the short window the frequencies are blurred but each change lands at its exact moment. With the long window the frequencies are sharp and the change is smeared in time.
Uses the setup above, 30 lines.
import matplotlib.pyplot as plt t = np.arange(0, 1.0, 1 / rate) x = np.where(t < 0.5, np.sin(2 * np.pi * 500 * t), np.sin(2 * np.pi * 1500 * t)) fig, axes = plt.subplots(3, 1, figsize=(5, 6)) axes[0].plot(t[:1200], x[:1200]) axes[0].set_title("waveform (first 0.15 s)") for ax, size in zip(axes[1:], (64, 512)): S = stft(x, size, size // 4) ax.imshow(S.T, origin="lower", aspect="auto", extent=[0, 1, 0, rate / 2]) ax.set_title(f"window of {size} samples ({1000 * size // rate} ms)") ax.set_ylabel("Hz") axes[2].set_xlabel("seconds") plt.tight_layout() plt.show()
Two bright bands appear: low in the first half, high in the second. Read the image across for time and up for pitch.
What a speech model is given
A spectrogram still has more detail than speech needs, in a particular place. People tell 300 Hz from 350 Hz easily and can't tell 6,000 from 6,050 at all: hearing is fine-grained at low pitches and coarse at high ones. The mel scale is a measure of pitch that follows that. Equal steps in mels are small steps in hertz at the bottom and big ones at the top.
A mel spectrogram adds up the spectrogram's energy inside a few dozen overlapping bands, spaced evenly in mels. Taking the logarithm of each sum turns energy into something like loudness.
Uses the setup above, 30 lines.
import numpy as np rate = 8000 bank, edges = mel_filters(20, 256, rate) print("band edges in Hz:", np.round(edges[:6]).astype(int).tolist(), "...", np.round(edges[-4:]).astype(int).tolist()) print("the first band is", round(float(edges[1] - edges[0])), "Hz wide, and the last is", round(float(edges[-1] - edges[-2])), "Hz wide") t = np.arange(0, 1.0, 1 / rate) x = np.where(t < 0.5, np.sin(2 * np.pi * 500 * t), np.sin(2 * np.pi * 1500 * t)) S = stft(x, 256, 128) log_mel = np.log(S ** 2 @ bank.T + 1e-6) # energy in each band, on a log scale print() print("one second of sound:", len(x), "samples -> spectrogram", S.shape, "=", S.size, "numbers -> log-mel", log_mel.shape, "=", log_mel.size, "numbers") print("loudest band, first half:", int(log_mel[10].argmax()), "| second half:", int(log_mel[50].argmax()))
The bands start 66 Hz wide and end 407 Hz wide. The 129 frequencies of each frame become 20 numbers. One second of sound, 8,000 samples, is now 61 rows of 20: 1,220 numbers, and the change of pitch at half-time is still plain to see, in band 5 and then band 12.
This is the log-mel spectrogram, the usual input to a speech recognizer. It keeps what the ear keeps: which bands are loud, and how that changes every few milliseconds. It throws away exact pitch at high frequencies, and one more thing called phase, the precise alignment of each wave, which carries little of the meaning of speech.
That last point matters when going the other way. A model that makes speech produces a mel spectrogram first, and turning that back into a waveform means inventing the missing detail. The part that does it is called a vocoder, and it is a model of its own.
Show the answer
Hearing tells low pitches apart finely and high ones coarsely
The mel scale follows perception: a 50 Hz step is obvious near 300 Hz and inaudible near 6,000 Hz. Speech is made for ears, so the detail that distinguishes speech sounds is spread the same way. Spending the bands where the differences are heard keeps what matters in far fewer numbers.
seconds, rate, bands = 30, 16000, 80 samples = seconds * rate rows = seconds * 100 print(samples, rows, samples / (rows * bands))
Show the answer
480000 3000 2.0
Thirty seconds is 480,000 samples. A row every 10 ms is 100 rows a second, so 3,000 rows, each of 80 numbers: 240,000 numbers. That is only half as many numbers as the samples. The gain isn't mainly in size. It is that the 3,000 rows are a short sequence a transformer can attend over, where 480,000 samples would not be.
Worked example: the features for three seconds of sound
Here is the whole front end of a speech recognizer in one function: frames of 25 ms every 10 ms, 40 mel bands, a log. It is run on three seconds of sound at 16,000 samples a second: a tone that rises for one second, a second of silence, then a quieter steady tone.
Uses the setup above, 30 lines.
import numpy as np rate = 8000 def features(x, rate, window_ms=25, hop_ms=10, bands=40): """What a speech recognizer is given: a log-mel spectrogram, one row for every 10 ms of sound.""" size, hop = int(rate * window_ms / 1000), int(rate * hop_ms / 1000) bank, edges = mel_filters(bands, size, rate) return np.log(stft(x, size, hop) ** 2 @ bank.T + 1e-6) rate = 16000 # three seconds: a rising tone, then silence, then a steady one t = np.arange(0, 3.0, 1 / rate) x = (np.where(t < 1.0, np.sin(2 * np.pi * (300 * t + 400 * t ** 2)), 0.0) + np.where(t >= 2.0, 0.3 * np.sin(2 * np.pi * 2500 * t), 0.0)) F = features(x, rate) print("samples:", len(x), "| features:", F.shape, "| rows per second:", round(F.shape[0] / 3)) for seconds in (0.1, 0.5, 0.9, 1.5, 2.5): row = F[int(seconds * 100)] print(" at", seconds, "s | loudest band:", str(int(row.argmax())).rjust(2), "| its level:", f"{row.max():6.1f}", "| quiet" if row.max() < 0 else "") print("thirty seconds of sound would be", 30 * 100, "rows of", F.shape[1], "numbers:", 30 * 100 * F.shape[1], "numbers in place of", f"{30 * rate:,}", "samples")
The 48,000 samples become 298 rows of 40 numbers, about 100 rows a second. Reading down the rows tells the story of the sound. During the first second the loudest band climbs from 6 to 10 to 14, the rising tone. At 1.5 seconds the level is −13.8: on a log scale, silence shows up as a large negative number, which is how a recognizer knows nobody is speaking. At 2.5 seconds the loudest band is 24, a high steady tone, and its level of 6.9 against 9.2 earlier says it is quieter. The last line is the scale a real system works at: thirty seconds of sound is 3,000 rows. Every later stage of a speech model, the convolutions, the transformer, the decoder that writes text, works on those rows and never sees a sample.
Write a sentence or two, then compare it with a good answer in the lesson.
In real models: the front end of a speech model
The function you built is what a library call does. Real speech models take a log-mel array like this and hand it to a network. Its rows are the sequence, so the model sees a list of 298 vectors of 40 numbers, much as a text model sees a list of token vectors.
On the job: most speech bugs sit in the front end. Check the sampling rate first: audio recorded at 44,100 Hz and fed to a model trained at 16,000 sounds slowed down and gets garbage text. Resample, then check the feature shape against what the model expects.
Write a sentence or two, then compare it with a good answer in the lesson.
Goes deeper in
- Recognition and synthesis as a pipeline: an encoder-decoder over these log-mel frames.
- The voice agent loop and its latency budget: the loop of listening, thinking and speaking, and its time budget.
Practice what you just read
- 7 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 13 review cards for spaced repetition
Try a quiz question
A recording is made at 16,000 samples a second. What is the highest frequency it can hold correctly?
Before this lesson
The math behind this
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 5 · about 36 hours of study
Transformers
Attention, GPT from scratch and pretraining
47 lessons 1 free
- Language modeling FreePredict the next token, and score it with cross-entropy.30 min
- Tokenization and BPEHow tokenizers are built from data.40 min
- A bigram modelThe simplest language model, built from counts.35 min
- Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
- Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
- Neural language modelsWhy counting fails, and what embeddings fix.35 min
- Attention: the intuitionEach token decides which other tokens matter to it.30 min
- Queries, keys and valuesThe three projections behind attention.35 min
- Scaled dot-product attentionThe attention formula, line by line.35 min
- Causal maskingStop tokens from seeing the future.25 min
- Multi-head attentionSeveral attention patterns in parallel.40 min
- Positional encodingsTell the model where each token sits.30 min
- The transformer blockAttention, MLP, residuals and norms together.40 min
- The GPT architectureA stack of decoder blocks that predicts the next token.40 min
- Build a tiny GPTWrite and train a small GPT from scratch.45 min
- From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
- Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
- Encoders and embedding modelsBERT-style models and where they fit.40 min
- Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
- Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
- Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
- The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
- The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
- Pretraining at scaleData, compute and what models learn from the internet.40 min
- Scaling lawsHow loss falls predictably with size, data and compute.40 min
- Inference and the KV cacheWhy generation is fast after the first token.40 min
- The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
- Long contextEfficient attention and million-token windows.35 min
- Mixture of expertsRoute each token to a few specialist sub-networks.35 min
- Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
- Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
- Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
- The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
- Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
- Multimodal transformersHow models see images and hear audio.35 min
- Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
- Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
- Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
- Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
- Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
- Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
- Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
- Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
- Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
- Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
- Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
- The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write