Index Ai Academy
Phase 5 · Beyond text

Sound as numbers: waveforms and spectrograms

A recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.

About 40 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

Speech recognition, voice assistants and music models all start in the same place: a microphone turns moving air into a long list of numbers. A neural network could be handed that list directly, and a few are. Most are handed a picture of it instead, a spectrogram, for the same reason a vision model works on pixels in a grid: the structure that matters is easier to see. This lesson builds that picture from a raw waveform in a few lines of NumPy, so that "the audio encoder takes log-mel features" stops being a phrase and becomes an array whose shape you can predict.

What you'll be able to do

  • Describe a recording as an array of samples, and work out its size from the sampling rate
  • Explain why a sound must be sampled at more than twice its highest frequency, and what goes wrong otherwise
  • Turn a waveform into a spectrogram, and read which frequencies were present when
  • Choose a window length, knowing what it trades between pitch and timing
  • Build the log-mel features a speech model is given, and say why they are shaped that way
Quick check
A guess before you start. A microphone records one second of speech as 16,000 numbers. What does each number measure?
  1. The letter that is being spoken
  2. The loudness of one whole word
  3. The air pressure at one instant
  4. The pitch of the voice just then
Show the answer

The air pressure at one instant

Sound is air pressure rising and falling. A microphone measures that pressure thousands of times a second, and each measurement is one number, called a sample. Loudness and pitch aren't stored anywhere. They are patterns across many samples: how far the numbers swing, and how quickly.

The idea: a sound is a list of numbers

This rests on Waves: height, speed and shift, where a wave has a frequency and a period, and on Sine and cosine. Here the wave is air pressure.

A sound is air pressure wobbling. A microphone reads the pressure at regular moments, and each reading is a sample. The number of readings per second is the sampling rate, measured in hertz (Hz). The whole recording is one array, a waveform.

Setup (runs before every cell) · 30 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import numpy as np

rate = 8000

def stft(x, size, hop):
    """Cut x into overlapping frames of size samples, hop apart, and
    take each frame's spectrum."""
    window = np.hanning(size)                       # fades each frame in and out
    frames = np.array(
        [x[i : i + size] * window for i in range(0, len(x) - size + 1, hop)])
    return np.abs(np.fft.rfft(frames, axis=1))      # rows: frames. columns: frequencies

def hz_to_mel(f):
    return 2595 * np.log10(1 + f / 700)

def mel_to_hz(m):
    return 700 * (10 ** (m / 2595) - 1)

def mel_filters(count, size, rate):
    """count overlapping triangles across the spectrum: narrow at low
    frequencies, wide at high ones."""
    edges = mel_to_hz(np.linspace(0, hz_to_mel(rate / 2), count + 2))
    bins = np.fft.rfftfreq(size, 1 / rate)
    bank = np.zeros((count, len(bins)))
    for m in range(count):
        low, middle, high = edges[m], edges[m + 1], edges[m + 2]
        bank[m] = np.clip(
            np.minimum((bins - low) / (middle - low), (high - bins) / (high - middle)),
            0, None)
    return bank, edges
import numpy as np

rate = 8000                         # samples per second
t = np.arange(0, 0.01, 1 / rate)    # 10 ms of moments at which the sound is measured
tone = np.sin(2 * np.pi * 440 * t)  # 440 Hz: the air pressure swings 440 times a second
print("samples in 10 ms:", len(tone), "| first five:", np.round(tone[:5], 3).tolist())
print("one second at 16,000 samples a second:", 16000, "numbers | a minute:",
      f"{16000 * 60:,}", "| an hour:", f"{16000 * 3600:,}")
print()
for amplitude in (1.0, 0.5, 0.1, 0.01):
    x = amplitude * np.sin(2 * np.pi * 440 * np.arange(0, 1, 1 / rate))
    rms = np.sqrt((x ** 2).mean())  # root mean square: the average size of the swings
    print("amplitude", str(amplitude).ljust(4), "| rms", f"{rms:.4f}", "| in decibels:",
          f"{20 * np.log10(rms):6.1f}")
It runs right here in your browser, with nothing to install.

Ten milliseconds of sound is 80 numbers at this rate. The first five trace the start of a sine wave rising from zero: a pure tone is the wave you met in the math lessons, with time on the horizontal axis.

Speech models usually work at 16,000 samples a second. An hour of that is 57,600,000 numbers, which is why nobody feeds raw audio to a model without shrinking it first.

The second table is about loudness. Halving the amplitude halves the rms, the typical size of a sample. Ears and engineers both measure loudness on a log scale, in decibels: each halving is 6 dB down, and each division by ten is 20 dB down, as the last column shows. A whisper and a shout are far apart in amplitude and only a few tens of decibels apart.

Quick check
From the lesson on angles and waves: a wave repeats every 0.002 seconds. What is its frequency?
  1. 500 cycles a second
  2. 2 cycles a second
  3. 2,000 cycles a second
  4. 0.002 cycles a second
Show the answer

500 cycles a second

Frequency is 1 ÷ period. One cycle every 0.002 seconds is 1 ÷ 0.002 = 500 cycles a second, or 500 Hz. In sound, frequency is heard as pitch: 440 Hz is the note an orchestra tunes to, and speech lives mostly between about 100 and 4,000 Hz.

Quick check
A recording is 2.5 seconds long at 16,000 samples a second. What does this print: its length as an array, and how many samples one cycle of a 200 Hz hum takes?
rate, seconds, hum = 16000, 2.5, 200
print(int(rate * seconds), rate // hum)
  1. 6400 80
  2. 16000 200
  3. 40000 80
  4. 40000 200
Show the answer

40000 80

Samples are rate times seconds: 16,000 × 2.5 = 40,000. A 200 Hz hum repeats 200 times a second, and a second is 16,000 samples, so one cycle is 16,000 ÷ 200 = 80 samples. The lower the pitch, the more samples each cycle takes.

In your own words
A recording is stored as one number per sample. Where in those numbers is the pitch of a voice?

Write a sentence or two, then compare it with a good answer in the lesson.

How often to measure

How many samples a second are enough? The cell records pure tones of different pitches, always at 8,000 samples a second, and then asks which frequency the recording contains.

Uses the setup above, 30 lines.

import numpy as np

rate = 8000

def loudest_hz(x, rate):
    """The frequency with the most energy in x."""
    return float(np.fft.rfftfreq(len(x), 1 / rate)[np.abs(np.fft.rfft(x)).argmax()])

for f in (500, 1000, 3000, 3900, 5000, 7000, 7500):
    # one second of a tone at f Hz, sampled 8,000 times
    x = np.sin(2 * np.pi * f * np.arange(0, 1, 1 / rate))
    print("a tone of", str(f).rjust(4),
          "Hz, sampled 8,000 times a second, comes out as",
          str(round(loudest_hz(x, rate))).rjust(4), "Hz")

Up to 3,900 Hz the recording is faithful. Then it goes wrong in a specific way. A tone at 5,000 Hz comes out as 3,000. One at 7,000 comes out as 1,000. The recording isn't noisy or blurred. It contains a clean tone at the wrong pitch.

This is aliasing. With 8,000 samples a second, a 7,000 Hz wave moves so far between one sample and the next that the samples land exactly where a 1,000 Hz wave would have put them. From the numbers alone, the two can't be told apart.

The limit is half the sampling rate, called the Nyquist frequency: 4,000 Hz here. Anything above it folds back down, as far below the limit as it was above. So the rule is: sample at more than twice the highest frequency you want to keep, and filter out everything above half the rate before sampling.

That rule explains the standard rates. Telephones use 8,000 Hz, keeping speech up to 4,000 Hz: understandable, a little dull. Speech models use 16,000. Music uses 44,100 or more, since people hear up to about 20,000 Hz.

Drill

What pitch does the recording hold?

Write apparent_hz(f, rate): the frequency a pure tone of f Hz appears to have after being sampled at rate samples a second. Take the remainder of f divided by rate. If that is more than half the rate, the tone has folded: the answer is rate minus the remainder.

Uses the setup above, 30 lines.

def apparent_hz(f, rate):
    return f


for f in (1000, 5000, 7000):
    print(f, "->", apparent_hz(f, 8000))
In the lesson, your code is checked as you write it.
Try it now

Press Next. The shapes are the ones the worked example at the end of part 2 prints.

In your own words
Why is aliasing worse than simply losing the high frequencies?

Write a sentence or two, then compare it with a good answer in the lesson.

In PyTorch: from waveform to spectrogram

Which frequencies, and when

To find the frequencies in a stretch of samples there is a standard tool, the Fourier transform (np.fft.rfft). It takes a list of samples and returns how much of each frequency they contain: a spectrum.

Applied to a whole recording, it has a blind spot.

Uses the setup above, 30 lines.

import numpy as np

rate = 8000

t = np.arange(0, 1.0, 1 / rate)
# 500 Hz for half a second, then 1,500 Hz
x = np.where(t < 0.5, np.sin(2 * np.pi * 500 * t), np.sin(2 * np.pi * 1500 * t))

whole = np.abs(np.fft.rfft(x))
strongest = np.fft.rfftfreq(len(x), 1 / rate)[np.argsort(whole)[-2:]]
print("spectrum of the whole second: strongest at",
      sorted(np.round(strongest).astype(int).tolist()), "Hz. When? It can't say.")

S = stft(x, 256, 128)
print("spectrogram:", S.shape[0], "frames by", S.shape[1],
      "frequencies | a frame every", 1000 * 128 / rate, "ms | frequencies", rate / 256,
      "Hz apart")
for frame in (5, 20, 40, 55):
    print("  frame", str(frame).rjust(2), "at", f"{frame * 128 / rate:.2f}",
          "s | loudest:", round(float(S[frame].argmax() * rate / 256)), "Hz")

The sound is 500 Hz for half a second and then 1,500 Hz. The spectrum of the whole second finds both frequencies and has no idea of the order. For speech, order is everything.

The fix is to cut the recording into short overlapping frames and take the spectrum of each. Stack the results and you have a spectrogram: a grid with time along one side and frequency along the other. Here it has 61 frames and 129 frequencies, and it reads the sound correctly: 500 Hz early, 1,500 Hz late.

How long should a frame be? There is no free choice.

Uses the setup above, 30 lines.

import numpy as np

rate = 8000

t = np.arange(0, 1.0, 1 / rate)
# two tones only 60 Hz apart, for the whole second
x = np.sin(2 * np.pi * 1000 * t) + np.sin(2 * np.pi * 1060 * t)
x[4000:4008] += 4.0                                    # and a click 1 ms long, at 0.5 s

for size in (64, 256, 1024):
    hop = size // 4
    S = stft(x, size, hop)
    bins = np.fft.rfftfreq(size, 1 / rate)
    # the average spectrum, well before the click
    average = S[: len(S) // 3].mean(axis=0)
    band = np.where((bins >= 800) & (bins <= 1300))[0]
    tones = sum(1 for i in band
                if average[i] > average[i - 1] and average[i] >= average[i + 1]
                and average[i] > 0.5 * average[band].max())
    high = S[:, bins > 2000].sum(axis=1)  # only the click reaches up here
    touched = int((high > 0.2 * high.max()).sum())
    print("window of", str(size).rjust(4), "samples =",
          str(1000 * size // rate).rjust(3), "ms | frequencies",
          str(round(rate / size, 1)).rjust(5), "Hz apart | tones it can tell apart:",
          tones, "| the 1 ms click is spread over",
          str(round(1000 * (touched * hop + size - hop) / rate)).rjust(3), "ms")

This sound has two tones only 60 Hz apart, and a click 1 ms long. An 8 ms window pins the click down to 12 ms and sees the two tones as one. A 128 ms window separates the tones, with frequencies 7.8 Hz apart, and smears the click over 192 ms.

A short frame knows when and is vague about which pitch. A long frame is the reverse. It isn't a flaw in the method: a pitch takes time to exist, so a short stretch of sound doesn't have a precise one. Speech work settles on windows of 20 to 25 ms, moved along 10 ms at a time.

Quick check
A spectrogram is made from one second of sound at 8,000 samples a second, with frames of 400 samples that start 160 samples apart. What does this print: the number of frames, and the gap between frequencies in Hz?
samples, size, hop, rate = 8000, 400, 160, 8000
print((samples - size) // hop + 1, rate / size)
  1. 48 20.0
  2. 48 50.0
  3. 50 20.0
  4. 20 50.0
Show the answer

48 20.0

The first frame starts at sample 0, and each next one 160 later, as long as a whole frame of 400 still fits: (8,000 − 400) // 160 + 1 = 48 frames. A frame of 400 samples at 8,000 a second lasts 50 ms, and its spectrum has frequencies 8,000 ÷ 400 = 20 Hz apart. Longer frames give finer frequencies.

Drill

How many frames?

Write frame_count(samples, size, hop): how many whole frames of size samples fit in a recording of samples samples, when each frame starts hop samples after the last. A recording shorter than one frame has 0.

Uses the setup above, 30 lines.

def frame_count(samples, size, hop):
    return 0


print(frame_count(8000, 256, 128), frame_count(48000, 400, 160))
In the lesson, your code is checked as you write it.
In your own words
Why can't a spectrogram be sharp in both time and frequency at once?

Write a sentence or two, then compare it with a good answer in the lesson.

The spectrogram as a picture

Printing frames hides what a spectrogram is: a picture. The cell draws the waveform above its spectrogram for two window lengths. The sound is 500 Hz, then 1,500 Hz. With the short window the frequencies are blurred but each change lands at its exact moment. With the long window the frequencies are sharp and the change is smeared in time.

Uses the setup above, 30 lines.

import matplotlib.pyplot as plt

t = np.arange(0, 1.0, 1 / rate)
x = np.where(t < 0.5, np.sin(2 * np.pi * 500 * t), np.sin(2 * np.pi * 1500 * t))

fig, axes = plt.subplots(3, 1, figsize=(5, 6))
axes[0].plot(t[:1200], x[:1200])
axes[0].set_title("waveform (first 0.15 s)")
for ax, size in zip(axes[1:], (64, 512)):
    S = stft(x, size, size // 4)
    ax.imshow(S.T, origin="lower", aspect="auto", extent=[0, 1, 0, rate / 2])
    ax.set_title(f"window of {size} samples ({1000 * size // rate} ms)")
    ax.set_ylabel("Hz")
axes[2].set_xlabel("seconds")
plt.tight_layout()
plt.show()

Two bright bands appear: low in the first half, high in the second. Read the image across for time and up for pitch.

What a speech model is given

A spectrogram still has more detail than speech needs, in a particular place. People tell 300 Hz from 350 Hz easily and can't tell 6,000 from 6,050 at all: hearing is fine-grained at low pitches and coarse at high ones. The mel scale is a measure of pitch that follows that. Equal steps in mels are small steps in hertz at the bottom and big ones at the top.

A mel spectrogram adds up the spectrogram's energy inside a few dozen overlapping bands, spaced evenly in mels. Taking the logarithm of each sum turns energy into something like loudness.

Uses the setup above, 30 lines.

import numpy as np

rate = 8000

bank, edges = mel_filters(20, 256, rate)
print("band edges in Hz:", np.round(edges[:6]).astype(int).tolist(), "...",
      np.round(edges[-4:]).astype(int).tolist())
print("the first band is", round(float(edges[1] - edges[0])),
      "Hz wide, and the last is", round(float(edges[-1] - edges[-2])), "Hz wide")

t = np.arange(0, 1.0, 1 / rate)
x = np.where(t < 0.5, np.sin(2 * np.pi * 500 * t), np.sin(2 * np.pi * 1500 * t))
S = stft(x, 256, 128)
log_mel = np.log(S ** 2 @ bank.T + 1e-6)           # energy in each band, on a log scale
print()
print("one second of sound:", len(x), "samples -> spectrogram", S.shape, "=", S.size,
      "numbers -> log-mel", log_mel.shape, "=", log_mel.size, "numbers")
print("loudest band, first half:", int(log_mel[10].argmax()), "| second half:",
      int(log_mel[50].argmax()))

The bands start 66 Hz wide and end 407 Hz wide. The 129 frequencies of each frame become 20 numbers. One second of sound, 8,000 samples, is now 61 rows of 20: 1,220 numbers, and the change of pitch at half-time is still plain to see, in band 5 and then band 12.

This is the log-mel spectrogram, the usual input to a speech recognizer. It keeps what the ear keeps: which bands are loud, and how that changes every few milliseconds. It throws away exact pitch at high frequencies, and one more thing called phase, the precise alignment of each wave, which carries little of the meaning of speech.

That last point matters when going the other way. A model that makes speech produces a mel spectrogram first, and turning that back into a waveform means inventing the missing detail. The part that does it is called a vocoder, and it is a model of its own.

Quick check
Why are the mel bands narrow at low frequencies and wide at high ones?
  1. Hearing tells low pitches apart finely and high ones coarsely
  2. Low frequencies carry more energy, so they need more bands
  3. High frequencies are removed by the sampling rate anyway
  4. Narrow bands are faster to compute at low frequencies
Show the answer

Hearing tells low pitches apart finely and high ones coarsely

The mel scale follows perception: a 50 Hz step is obvious near 300 Hz and inaudible near 6,000 Hz. Speech is made for ears, so the detail that distinguishes speech sounds is spread the same way. Spending the bands where the differences are heard keeps what matters in far fewer numbers.

Quick check
A recognizer takes 30 seconds of sound at 16,000 samples a second and turns it into log-mel features: one row every 10 ms, 80 bands in each. What does this print: the samples, the rows, and how many times fewer numbers the features are?
seconds, rate, bands = 30, 16000, 80
samples = seconds * rate
rows = seconds * 100
print(samples, rows, samples / (rows * bands))
  1. 16000 3000 2.0
  2. 480000 3000 160.0
  3. 480000 3000 2.0
  4. 480000 300 20.0
Show the answer

480000 3000 2.0

Thirty seconds is 480,000 samples. A row every 10 ms is 100 rows a second, so 3,000 rows, each of 80 numbers: 240,000 numbers. That is only half as many numbers as the samples. The gain isn't mainly in size. It is that the 3,000 rows are a short sequence a transformer can attend over, where 480,000 samples would not be.

Worked example

Worked example: the features for three seconds of sound

Here is the whole front end of a speech recognizer in one function: frames of 25 ms every 10 ms, 40 mel bands, a log. It is run on three seconds of sound at 16,000 samples a second: a tone that rises for one second, a second of silence, then a quieter steady tone.

Uses the setup above, 30 lines.

import numpy as np

rate = 8000

def features(x, rate, window_ms=25, hop_ms=10, bands=40):
    """What a speech recognizer is given: a log-mel spectrogram, one row
    for every 10 ms of sound."""
    size, hop = int(rate * window_ms / 1000), int(rate * hop_ms / 1000)
    bank, edges = mel_filters(bands, size, rate)
    return np.log(stft(x, size, hop) ** 2 @ bank.T + 1e-6)

rate = 16000
# three seconds: a rising tone, then silence, then a steady one
t = np.arange(0, 3.0, 1 / rate)
x = (np.where(t < 1.0, np.sin(2 * np.pi * (300 * t + 400 * t ** 2)), 0.0)
     + np.where(t >= 2.0, 0.3 * np.sin(2 * np.pi * 2500 * t), 0.0))

F = features(x, rate)
print("samples:", len(x), "| features:", F.shape, "| rows per second:",
      round(F.shape[0] / 3))
for seconds in (0.1, 0.5, 0.9, 1.5, 2.5):
    row = F[int(seconds * 100)]
    print(" at", seconds, "s | loudest band:", str(int(row.argmax())).rjust(2),
          "| its level:", f"{row.max():6.1f}", "| quiet" if row.max() < 0 else "")
print("thirty seconds of sound would be", 30 * 100, "rows of", F.shape[1], "numbers:",
      30 * 100 * F.shape[1], "numbers in place of", f"{30 * rate:,}", "samples")
In the lesson you can change this and run it.

The 48,000 samples become 298 rows of 40 numbers, about 100 rows a second. Reading down the rows tells the story of the sound. During the first second the loudest band climbs from 6 to 10 to 14, the rising tone. At 1.5 seconds the level is −13.8: on a log scale, silence shows up as a large negative number, which is how a recognizer knows nobody is speaking. At 2.5 seconds the loudest band is 24, a high steady tone, and its level of 6.9 against 9.2 earlier says it is quieter. The last line is the scale a real system works at: thirty seconds of sound is 3,000 rows. Every later stage of a speech model, the convolutions, the transformer, the decoder that writes text, works on those rows and never sees a sample.

In your own words
Describe the steps from a microphone to the array a speech recognizer takes as input, and what each step is for.

Write a sentence or two, then compare it with a good answer in the lesson.

In real models: the front end of a speech model

The function you built is what a library call does. Real speech models take a log-mel array like this and hand it to a network. Its rows are the sequence, so the model sees a list of 298 vectors of 40 numbers, much as a text model sees a list of token vectors.

On the job: most speech bugs sit in the front end. Check the sampling rate first: audio recorded at 44,100 Hz and fed to a model trained at 16,000 sounds slowed down and gets garbage text. Resample, then check the feature shape against what the model expects.

In your own words
What is the first thing to check when a speech model gives garbage text on a new recording?

Write a sentence or two, then compare it with a good answer in the lesson.

Goes deeper in

Practice what you just read

  • 7 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 13 review cards for spaced repetition

Try a quiz question

A recording is made at 16,000 samples a second. What is the highest frequency it can hold correctly?

  1. Just under 8,000 Hz
  2. Just under 16,000 Hz
  3. Just under 32,000 Hz
  4. Just under 4,000 Hz

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

The math behind this

The Phase 2 lessons this one rests on, from the course map.

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 5 · about 36 hours of study

Transformers

Attention, GPT from scratch and pretraining 47 lessons 1 free
  1. Language modeling FreePredict the next token, and score it with cross-entropy.30 min
  2. Tokenization and BPEHow tokenizers are built from data.40 min
  3. A bigram modelThe simplest language model, built from counts.35 min
  4. Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
  5. Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
  6. Neural language modelsWhy counting fails, and what embeddings fix.35 min
  7. Attention: the intuitionEach token decides which other tokens matter to it.30 min
  8. Queries, keys and valuesThe three projections behind attention.35 min
  9. Scaled dot-product attentionThe attention formula, line by line.35 min
  10. Causal maskingStop tokens from seeing the future.25 min
  11. Multi-head attentionSeveral attention patterns in parallel.40 min
  12. Positional encodingsTell the model where each token sits.30 min
  13. The transformer blockAttention, MLP, residuals and norms together.40 min
  14. The GPT architectureA stack of decoder blocks that predicts the next token.40 min
  15. Build a tiny GPTWrite and train a small GPT from scratch.45 min
  16. From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
  17. Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
  18. Encoders and embedding modelsBERT-style models and where they fit.40 min
  19. Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
  20. Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
  21. Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
  22. The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
  23. The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
  24. Pretraining at scaleData, compute and what models learn from the internet.40 min
  25. Scaling lawsHow loss falls predictably with size, data and compute.40 min
  26. Inference and the KV cacheWhy generation is fast after the first token.40 min
  27. The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
  28. Long contextEfficient attention and million-token windows.35 min
  29. Mixture of expertsRoute each token to a few specialist sub-networks.35 min
  30. Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
  31. Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
  32. Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
  33. The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
  34. Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
  35. Multimodal transformersHow models see images and hear audio.35 min
  36. Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
  37. Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
  38. Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
  39. Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
  40. Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
  41. Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
  42. Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
  43. Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
  44. Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
  45. Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
  46. Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
  47. The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write