Image-text models in depth: retrieval, negatives and limits
How to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".
What it covers6
Why it matters
In the lesson on multimodal transformers you trained a small CLIP: two encoders pulled matching images and captions together until they shared one space. That space now sits under image search, zero-shot classifiers and the image side of most vision-language models. This lesson asks the questions an engineer has to ask before relying on it. How do you measure whether the space is any good? What decides how fine its distinctions are? And what can a single similarity score never tell you, however well it is trained?
What you'll be able to do
- Test a shared image-text space by retrieval, and report recall at k in both directions
- Explain what the other pairs in a batch do, and why a larger batch teaches finer distinctions
- Say what happens when every negative is hard, and what a false negative is
- Describe what the temperature does to the loss
- Show why a score built from summed word vectors can't see word order, and name tasks that follow from it
Show the answer
It may score both about the same, since they use the same words
The two captions contain exactly the same words. A model whose text side works mostly from which words are present, and less from how they are arranged, sees them as nearly the same sentence. Large image-text models have been measured doing poorly on exactly this kind of test. The last part of this lesson shows where the weakness comes from.
The idea: a table of scores, and the test of it
This rests on Multimodal transformers, where the similarity table first appeared, and on Angle, cosine similarity and projection for the similarity itself.
Retrieval: the test of a shared space
Contrastive training gives each image and each caption a vector, placed so that a matching pair points the same way. The direct test of such a space is retrieval: take a caption, rank every image in a collection by similarity to it, and see where the right image lands.
The standard score is recall at k, written R@k: the share of queries whose right answer is among the top k results. R@1 asks for the exact match in first place. R@5 forgives a near miss.
This lesson's data is a little richer than last time's. There are 20 kinds of scene, and every scene also has details of its own, so two dog photos are similar and not the same. Telling a dog from a car is the coarse distinction. Telling this dog photo from that one is the fine one.
Setup (runs before every cell) · 67 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import math import numpy as np import torch import torch.nn as nn import torch.nn.functional as F g0 = np.random.default_rng(0) KINDS = 20 # 20 kinds of scene: dog, car, beach... centers = g0.normal(size=(KINDS, 4)).astype(np.float32) to_pixels, to_words = (g0.normal(size=(12, 40)).astype(np.float32), g0.normal(size=(12, 20)).astype(np.float32)) def pairs(n, seed): """n matching (image, caption) pairs. A scene is its kind (4 numbers) plus its own details (8 small numbers).""" g = np.random.default_rng(seed) kind = g.integers(0, KINDS, size=n) scene = np.concatenate( [centers[kind], 0.3 * g.normal(size=(n, 8)).astype(np.float32)], axis=1) images = scene @ to_pixels + 0.5 * g.normal(size=(n, 40)).astype(np.float32) captions = scene @ to_words + 0.5 * g.normal(size=(n, 20)).astype(np.float32) return images, captions, kind def unit(v): return v / v.norm(dim=-1, keepdim=True) def train(batch, steps=300, temperature=0.1, one_kind=False, seed=0): torch.manual_seed(seed) image_encoder, text_encoder = nn.Linear(40, 16), nn.Linear(20, 16) opt = torch.optim.Adam( list(image_encoder.parameters()) + list(text_encoder.parameters()), lr=1e-2) images, captions, kind = pairs(4096, 1) g = np.random.default_rng(seed) for step in range(steps): if one_kind: # every pair in the batch is the same kind of scene idx = g.choice(np.where(kind == g.integers(0, KINDS))[0], size=batch, replace=False) else: idx = g.permutation(4096)[:batch] scores = (unit(image_encoder(torch.tensor(images[idx]))) @ unit(text_encoder(torch.tensor(captions[idx]))).T / temperature) labels = torch.arange(batch) # image i goes with caption i loss = (F.cross_entropy(scores, labels) + F.cross_entropy(scores.T, labels)) / 2 opt.zero_grad() loss.backward() opt.step() return image_encoder, text_encoder def similarities(image_encoder, text_encoder): """500 unseen pairs: the similarity of every image (rows) with every caption (columns).""" images, captions, kind = pairs(500, 2) with torch.no_grad(): sims = (unit(image_encoder(torch.tensor(images))) @ unit(text_encoder(torch.tensor(captions))).T).numpy() return sims, kind def retrieval(image_encoder, text_encoder): """Search the 500 images with each caption. Returns recall@1, recall@5, and how often the top image is the right kind.""" sims, kind = similarities(image_encoder, text_encoder) # images ranked above the right one better = np.array([(sims[:, j] > sims[j, j]).sum() for j in range(500)]) right_kind = (kind[sims.argmax(axis=0)] == kind).mean() return (round(float((better == 0).mean()), 3), round(float((better < 5).mean()), 3), round(float(right_kind), 3))
torch.manual_seed(0) print("untrained | recall@1, recall@5, top image is the right kind:", retrieval(nn.Linear(40, 16), nn.Linear(20, 16))) print("trained | recall@1, recall@5, top image is the right kind:", retrieval(*train(64)))
Before training, a caption's right image is never in first place among 500, and almost never even the right kind. After 300 steps on batches of 64, the exact image comes first for 0.832 of captions and is in the top five for 0.974.
The third number separates the two kinds of mistake. For 0.948 of captions the top image is at least the right kind of scene. So most of the misses are near misses: a dog photo was wanted and a different dog photo came back. Whether that matters depends on the product. For "show me pictures of dogs" it is a success. For "find the photo this caption was written for" it is a failure.
Show the answer
Its own caption's score goes up, the others' go down
Each image is scored against every caption in the batch, and a cross-entropy loss treats its own caption as the right answer. Raising that score and lowering the rest is what moves matching pairs together and everything else apart. The same is done from each caption's side.
sims = [[0.9, 0.8, 0.1], [0.2, 0.7, 0.3], [0.4, 0.6, 0.5]] ranks = [1 + sum(sims[i][j] > sims[j][j] for i in range(3)) for j in range(3)] print(ranks, round(sum(r == 1 for r in ranks) / 3, 3))
Show the answer
[1, 2, 1] 0.667
For caption 1, read down column 1: 0.8, 0.7, 0.6. Its own image, image 1, scores 0.7, and image 0 scores higher, so the right image is in second place. Captions 0 and 2 have their own image on top. Two of three are exact: R@1 is 0.667.
Rank of the right image for each caption
Write match_ranks(sims). sims is a square list of rows: sims[i][j] is the similarity of image i with caption j, and image j is the right one for caption j. For each caption, return the position of its right image when all the images are ranked by similarity to that caption: 1 plus the number of images that score strictly higher than the right one. Return the list of positions, one per caption.
Uses the setup above, 67 lines.
def match_ranks(sims): return [] print(match_ranks([[0.9, 0.8, 0.1], [0.2, 0.7, 0.3], [0.4, 0.6, 0.5]]))
Press Next. Retrieval reads a column; the loss pushes the diagonal up and everything else down.
Write a sentence or two, then compare it with a good answer in the lesson.
In PyTorch: what the rest of the batch is for
In contrastive training, an image's own caption is its positive. Every other caption in the batch is a negative: a wrong answer the model must rank below the right one. With 64 pairs in a batch, each image faces a 64-way choice.
The negatives decide what the model has to learn. If the only wrong answers on offer are about cars and beaches, a dog photo can find its caption by knowing "dog", and the details never matter. If another dog caption is in the batch, "dog" isn't enough any more. So a bigger batch does more than average the gradient: it raises the odds that each pair meets a look-alike.
Uses the setup above, 67 lines.
import matplotlib.pyplot as plt print("pairs per batch | recall@1 | recall@5 | top image is the right kind") batches, recall1, recall5 = (4, 16, 64, 256), [], [] for batch in batches: r1, r5, right_kind = retrieval(*train(batch)) recall1.append(r1) recall5.append(r5) print(str(batch).rjust(15), "|", str(r1).ljust(8), "|", str(r5).ljust(8), "|", right_kind) plt.plot(batches, recall1, "o-", label="recall@1") plt.plot(batches, recall5, "s-", label="recall@5") plt.xscale("log") plt.xlabel("pairs per batch") plt.ylabel("recall") plt.legend() print("64, all of one kind:", retrieval(*train(64, one_kind=True)))
With 4 pairs per batch, R@1 is 0.436. With 256 it is 0.858. The gain is steep at first and flattens. With 20 kinds of scene, a batch of 4 almost never holds two of a kind, a batch of 64 holds about three of each, and by then most of what there is to learn from look-alikes is being learned. Real image-text models face millions of fine distinctions, and were trained with batches in the tens of thousands.
The last line is a warning against taking that too far. Negatives that resemble the positive are called hard negatives, and if look-alikes are what teach detail, a batch made only of one kind of scene sounds ideal. It scores 0.43, and its top image is the right kind for fewer than half the captions. A model that never sees a dog next to a car is never asked to tell them apart, and it doesn't.
One more hazard hides in the batch. Two photos of the same event may each fit the other's caption. The loss still treats the other caption as wrong and pushes it away. That is a false negative, and web-scale data is full of them.
Show the answer
127
The image's own caption is the one right answer. The other 127 captions in the batch are its negatives. No one chose them as wrong answers: they are simply the captions of the other images that happened to be in the batch.
Show the answer
It was never asked to tell one kind of scene from another
Every negative it met was the same kind of scene as the positive, so the loss only ever rewarded telling details apart. Nothing pushed dogs away from cars. At test time the collection holds every kind, and the model's coarse sense of kind is weak. Negatives need variety: some easy, some hard.
The other dial on the loss is the temperature, the small number the similarities are divided by before the softmax. You saw in Multimodal transformers that dividing by 0.1 stretches the scores apart. Here is what that does to training, and to a single comparison.
Uses the setup above, 67 lines.
for temperature in (1.0, 0.1, 0.02): print("temperature", str(temperature).ljust(4), "| recall@1, recall@5, right kind:", retrieval(*train(64, temperature=temperature))) print("two captions score 0.9 and 0.8 against an image. " "Probability given to the better one:") for temperature in (1.0, 0.1, 0.02): a, b = math.exp(0.9 / temperature), math.exp(0.8 / temperature) print(" temperature", str(temperature).ljust(4), "->", round(a / (a + b), 3))
At a temperature of 1.0, a caption scoring 0.9 gets a probability of 0.525 against one scoring 0.8. The model is already about as right as the loss can see, so there is little pressure to separate close candidates, and R@1 is 0.79. At 0.02 the same gap becomes 0.993 to 0.007: small differences in similarity now matter a great deal, and R@1 rises to 0.852.
A lower temperature makes the model care about its closest rivals, which is the same thing a larger batch buys by a different route. It isn't free in real training, where a very low temperature makes the loss jumpy and punishes false negatives hard, so models such as CLIP let training choose it.
import math for temperature in (1.0, 0.05): a, b = math.exp(0.6 / temperature), math.exp(0.5 / temperature) print(round(a / (a + b), 3), end=" ")
Show the answer
0.525 0.881
At a temperature of 1 the gap between the scores is 0.1, and the softmax splits 0.525 to 0.475. Dividing by 0.05 turns the gap into 2, and the split becomes 0.881 to 0.119. The scores didn't change. The temperature decides how much a small lead counts.
Write a sentence or two, then compare it with a good answer in the lesson.
In real models: what a similarity score can't say
Everything above ends in one number per image and caption. That number is good at "what is this about". It is weaker at "how are the parts arranged", and the reason can be shown without training anything.
Take the simplest possible text encoder: give each word a vector, and add up the vectors of the words in the caption. This is called a bag of words, since it keeps the words and throws away their order.
Uses the setup above, 67 lines.
import numpy as np g = np.random.default_rng(0) vocabulary = "a red blue square circle and dog street with no on the".split() vector = {word: g.normal(size=16) for word in vocabulary} # one vector per word def encode(caption): # add up the words' vectors: order is lost here total = sum(vector[word] for word in caption.split()) return total / np.linalg.norm(total) def similarity(a, b): return round(float(encode(a) @ encode(b)), 3) tests = [("a red square and a blue circle", "a blue square and a red circle"), ("a dog on the street", "the street on a dog"), ("a street with a dog", "a street with no dog"), ("a street with a dog", "a red square on the street")] for a, b in tests: print(similarity(a, b), "|", a, "|", b)
The first two pairs score 1.0. To this encoder, "a red square and a blue circle" and "a blue square and a red circle" are the same caption: the same words, added up in a different order, give the same sum. So do "a dog on the street" and "the street on a dog". Whatever image you show, the two captions get identical scores, and the model can't prefer the right one.
The third pair shows a quieter failure. "A street with a dog" and "a street with no dog" score 0.904. One word differs, and it reverses the meaning. For comparison, a caption about something else entirely scores 0.627.
Real text encoders are transformers, which can represent word order. The trouble is the training game. With random negatives, a caption almost never meets its own words in a different arrangement, so the model can win nearly every comparison by knowing which things are present. Tests built from reordered captions have found large image-text models scoring close to chance on them.
What follows for an engineer: use these models for what a bag of things can answer, and check before trusting them with which attribute belongs to which object, relations such as "on" or "left of", counts, and negation.
Show the answer
"a cat chasing a dog" and "a dog chasing a cat"
Those two captions contain the same five words, each the same number of times, so their sums are equal, though they describe opposite events. Each of the other pairs differs in at least one word, which changes the sum.
Captions a bag of words can't tell apart
Write same_bag(a, b). It returns True when the two captions are different sentences that a bag-of-words encoder would see as identical: they contain exactly the same words, each the same number of times, and the words are not in the same order. Compare words in lower case, and split on any whitespace. Two captions with the same words in the same order are the same sentence: return False.
Uses the setup above, 67 lines.
def same_bag(a, b): return False print(same_bag("a red square and a blue circle", "a blue square and a red circle"))
Worked example: what kind of mistakes does the model make?
R@1 says how often the top result is exactly right. Before deciding whether a model is good enough, look at what it returns when it is wrong. Here two models, trained with 4 and with 256 pairs per batch, each answer 500 captions, and every answer is put in one of three bins.
Uses the setup above, 67 lines.
import numpy as np import torch import torch.nn as nn import torch.nn.functional as F g0 = np.random.default_rng(0) KINDS = 20 # 20 kinds of scene: dog, car, beach... centers = g0.normal(size=(KINDS, 4)).astype(np.float32) to_pixels, to_words = (g0.normal(size=(12, 40)).astype(np.float32), g0.normal(size=(12, 20)).astype(np.float32)) def pairs(n, seed): """n matching (image, caption) pairs. A scene is its kind (4 numbers) plus its own details (8 small numbers).""" g = np.random.default_rng(seed) kind = g.integers(0, KINDS, size=n) scene = np.concatenate( [centers[kind], 0.3 * g.normal(size=(n, 8)).astype(np.float32)], axis=1) images = scene @ to_pixels + 0.5 * g.normal(size=(n, 40)).astype(np.float32) captions = scene @ to_words + 0.5 * g.normal(size=(n, 20)).astype(np.float32) return images, captions, kind def unit(v): return v / v.norm(dim=-1, keepdim=True) def train(batch, steps=300, temperature=0.1, one_kind=False, seed=0): torch.manual_seed(seed) image_encoder, text_encoder = nn.Linear(40, 16), nn.Linear(20, 16) opt = torch.optim.Adam( list(image_encoder.parameters()) + list(text_encoder.parameters()), lr=1e-2) images, captions, kind = pairs(4096, 1) g = np.random.default_rng(seed) for step in range(steps): if one_kind: # every pair in the batch is the same kind of scene idx = g.choice(np.where(kind == g.integers(0, KINDS))[0], size=batch, replace=False) else: idx = g.permutation(4096)[:batch] scores = (unit(image_encoder(torch.tensor(images[idx]))) @ unit(text_encoder(torch.tensor(captions[idx]))).T / temperature) labels = torch.arange(batch) # image i goes with caption i loss = (F.cross_entropy(scores, labels) + F.cross_entropy(scores.T, labels)) / 2 opt.zero_grad() loss.backward() opt.step() return image_encoder, text_encoder def similarities(image_encoder, text_encoder): """500 unseen pairs: the similarity of every image (rows) with every caption (columns).""" images, captions, kind = pairs(500, 2) with torch.no_grad(): sims = (unit(image_encoder(torch.tensor(images))) @ unit(text_encoder(torch.tensor(captions))).T).numpy() return sims, kind def retrieval(image_encoder, text_encoder): """Search the 500 images with each caption. Returns recall@1, recall@5, and how often the top image is the right kind.""" sims, kind = similarities(image_encoder, text_encoder) # images ranked above the right one better = np.array([(sims[:, j] > sims[j, j]).sum() for j in range(500)]) right_kind = (kind[sims.argmax(axis=0)] == kind).mean() return (round(float((better == 0).mean()), 3), round(float((better < 5).mean()), 3), round(float(right_kind), 3)) print("pairs per batch | exact match | right kind, wrong scene | wrong kind") for batch in (4, 256): sims, kind = similarities(*train(batch)) top = sims.argmax(axis=0) # the best image for each caption exact = int((top == np.arange(500)).sum()) near = int(((top != np.arange(500)) & (kind[top] == kind)).sum()) print(str(batch).rjust(15), "|", str(exact).rjust(11), "|", str(near).rjust(23), "|", 500 - exact - near)
The model trained on batches of 4 returns the exact image for 218 captions of 500. Its misses are mostly of one type: 220 times it returned the right kind of scene and the wrong one of that kind, against 62 times the wrong kind altogether. It learned kinds and not details, which is what a batch too small to contain look-alikes teaches. With 256 pairs per batch the exact matches rise to 429, and the near misses fall from 220 to 49, far more than the outright misses fall, from 62 to 22. The larger batch mostly bought fine distinctions. This breakdown also tells you whether a model fits a product: for a search box that only needs the right kind of picture, the first model is right 438 times in 500, and for matching a caption to its own photo it is right 218 times.
On the job: an image-text model is a retrieval engine first. To build search over a photo library, embed every photo once, store the vectors, and embed each query as it arrives, then rank by cosine similarity. Judge it with the numbers of this lesson, R@1 and R@5 on a set of your own queries with known right answers, and read the misses: near misses and wrong kinds need different fixes.
Write a sentence or two, then compare it with a good answer in the lesson.
Goes deeper in
- Self-supervised learning: labels from the data itself: learning from images alone, with the same pull-together, push-apart idea.
- Multimodal transformers: how an image-text model's vectors feed a language model.
- Tuning an embedding model for retrieval: fine-tuning embeddings for your own retrieval task.
Practice what you just read
- 5 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 13 review cards for spaced repetition
Try a quiz question
A caption-to-image search model reports R@1 = 0.40 and R@10 = 0.95. What does that combination suggest?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 5 · about 36 hours of study
Transformers
Attention, GPT from scratch and pretraining
47 lessons 1 free
- Language modeling FreePredict the next token, and score it with cross-entropy.30 min
- Tokenization and BPEHow tokenizers are built from data.40 min
- A bigram modelThe simplest language model, built from counts.35 min
- Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
- Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
- Neural language modelsWhy counting fails, and what embeddings fix.35 min
- Attention: the intuitionEach token decides which other tokens matter to it.30 min
- Queries, keys and valuesThe three projections behind attention.35 min
- Scaled dot-product attentionThe attention formula, line by line.35 min
- Causal maskingStop tokens from seeing the future.25 min
- Multi-head attentionSeveral attention patterns in parallel.40 min
- Positional encodingsTell the model where each token sits.30 min
- The transformer blockAttention, MLP, residuals and norms together.40 min
- The GPT architectureA stack of decoder blocks that predicts the next token.40 min
- Build a tiny GPTWrite and train a small GPT from scratch.45 min
- From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
- Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
- Encoders and embedding modelsBERT-style models and where they fit.40 min
- Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
- Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
- Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
- The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
- The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
- Pretraining at scaleData, compute and what models learn from the internet.40 min
- Scaling lawsHow loss falls predictably with size, data and compute.40 min
- Inference and the KV cacheWhy generation is fast after the first token.40 min
- The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
- Long contextEfficient attention and million-token windows.35 min
- Mixture of expertsRoute each token to a few specialist sub-networks.35 min
- Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
- Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
- Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
- The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
- Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
- Multimodal transformersHow models see images and hear audio.35 min
- Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
- Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
- Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
- Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
- Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
- Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
- Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
- Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
- Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
- Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
- Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
- The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write