Multimodal transformers

Connecting pictures and words: contrastive training

Picture a party game. You get a pile of photos and a pile of captions, all shuffled. Your job is to pair each photo with its caption. You get better by learning which words go with which sights.

OpenAI's CLIP (2021) learned in much the same way. It has two encoders. One turns an image into a vector. The other turns a caption into a vector. CLIP was trained on about 400 million image and caption pairs from the web.

The training game is called contrastive learning. Each batch holds some images and their captions. The model scores every image against every caption. Then training pushes each image's score with its own caption up. It pushes its scores with all the other captions down.

The score is cosine similarity. Each vector is first scaled to length 1. Then the dot product of two vectors says how closely they point the same way. The top score is 1, when the two vectors point exactly the same way.

Here's a batch of two, with made-up scores. Image 1 goes with caption A. Image 2 goes with caption B.

The right answers sit on the diagonal, from top left to bottom right. The grading is a cross-entropy loss, which you met in Phase 2. Each image is graded on how much probability it gives its own caption. The code below trains a small CLIP this way.

import math
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)
concepts = torch.randn(10, 6)                       # 10 hidden "things", shared by pictures and words
to_pixels, to_words = torch.randn(6, 40), torch.randn(6, 20)

def make_pairs(n):
    c = torch.randint(0, 10, (n,))
    z = concepts[c] + 0.3 * torch.randn(n, 6)
    images = z @ to_pixels + 0.5 * torch.randn(n, 40)     # stand-in images: 40 pixel features
    captions = z @ to_words + 0.5 * torch.randn(n, 20)    # stand-in captions: 20 word features
    return images, captions, c

image_encoder, text_encoder = nn.Linear(40, 16), nn.Linear(20, 16)
temperature = 0.1

def unit(v):
    return v / v.norm(dim=-1, keepdim=True)

def clip_loss(img, txt):
    logits = unit(img) @ unit(txt).T / temperature        # every image against every caption
    labels = torch.arange(len(img))                       # the matching pair is on the diagonal
    return (F.cross_entropy(logits, labels) + F.cross_entropy(logits.T, labels)) / 2

@torch.no_grad()
def zero_shot_accuracy():
    images, _, truth = make_pairs(500)
    label_texts = text_encoder(concepts @ to_words)       # one clean caption per concept
    guess = (unit(image_encoder(images)) @ unit(label_texts).T).argmax(dim=-1)
    return (guess == truth).float().mean().item()

print(f"zero-shot accuracy before training: {zero_shot_accuracy():.0%}")
opt = torch.optim.Adam(list(image_encoder.parameters()) + list(text_encoder.parameters()), lr=1e-2)
for step in range(201):
    images, captions, _ = make_pairs(32)
    loss = clip_loss(image_encoder(images), text_encoder(captions))
    opt.zero_grad()
    loss.backward()
    opt.step()
    if step % 50 == 0:
        print(f"step {step:>3}  loss {loss.item():.3f}")
print(f"zero-shot accuracy after training:  {zero_shot_accuracy():.0%}")

The images and captions here are simulated, to keep the example fast. The training is CLIP's.

• concepts holds 10 hidden "things". make_pairs builds fake images (40 numbers each) and fake captions (20 numbers each) from them, plus some noise.
• image_encoder and text_encoder each turn their input into 16 numbers. These two are what training changes.
• unit scales each vector to length 1.
• clip_loss builds the score table, every image against every caption, and divides it by temperature. It grades both ways: each image picks a caption, and each caption picks an image. The loss is the average of the two.
• The loop trains for 201 steps on batches of 32 pairs. It prints the loss every 50 steps.

After training, images and text live in one shared embedding space. A picture and its caption land close together. The test score climbs from chance, about 10% with 10 concepts, to over 90%.

The loss doesn't reach zero. Part of the reason: a batch of 32 drawn from only 10 concepts holds look-alikes. Two images of the same concept still count as "wrong" for each other's captions. Real web data has the same problem.

Zero-shot classification

A shared space gives you something for free. Say you want to sort photos into labels. Write one caption per label, such as "a photo of a tent" and "a photo of a jacket". Turn each caption into a vector. Then give each image the label whose caption is most similar.

This is zero-shot classification. "Zero-shot" means the model saw zero labeled examples for your task. You never trained it on photos tagged "tent". The zero_shot_accuracy test in the code above works this way. It writes one clean caption per concept and picks the closest.

CLIP-style encoders power image search, content moderation and duplicate detection. They're also the eyes of many vision-language models, which come next.

Vision-language models

How do you let a chatbot see? The common recipe, made popular by LLaVA in 2023, joins three parts.

1. A vision encoder, often CLIP-style, turns the image into patch vectors.
2. A small projector maps those vectors into the language model's embedding space. Afterwards, each one looks like a token embedding.
3. The language model receives the image tokens alongside the text tokens. It attends over all of them together.

The projector works like a travel plug adapter. The vision encoder's output has the wrong shape for the language model. The projector reshapes it so it fits.

Training usually starts with the projector alone, on image and caption pairs. Then more of the system is fine-tuned on instructions about images. Some newer models train on images and text together from the start. They're called "natively multimodal". The idea is the same: everything becomes a sequence of vectors in one model.

Audio works the same way. OpenAI's Whisper (2022) cuts sound into 30-second clips. It turns each clip into a spectrogram: a picture of how much energy each frequency has over time. Think of sheet music, which shows which notes sound at each moment. A transformer encoder reads the spectrogram, and a decoder writes the transcript. Whisper was trained on 680,000 hours of audio. Some speech-capable assistants feed audio tokens straight to the language model. Others pair the model with separate tools for speech recognition and speech synthesis.

Generating images is usually a different technique. Most image generators are diffusion models. They start from random noise and remove it step by step, steered by a text encoder. Some newer systems instead generate image tokens the way an LLM generates words.

In practice

Images cost tokens. Each image becomes hundreds to thousands of tokens, and you pay for tokens. Models merge patches, and providers publish their own counting rules. Anthropic's documentation has given an estimate of about (width × height) / 750 tokens. Large images are shrunk first.

width, height = 1000, 1000
patch = 16
patches = (width // patch) * (height // patch)
estimate = width * height / 750
print("whole 16-pixel patches:", patches)
print("estimated tokens:", round(estimate))

Fine detail is the weak spot. Small print, dense tables, exact counts and precise positions are harder than "what's in this picture". Detail can fall between or inside patches, and it gets compressed. Cropping to the part that matters helps. So does sending a higher-resolution crop.

Charts and screenshots are read, not measured. A model can describe a chart's trend well and still misread the exact values. When the numbers matter, give it the data too.

For marketers: vision models make it practical to tag product photos, check ads against brand guidelines, or pull text from screenshots of competitors' pages. Check any extracted numbers and small print before they go into a report.