Glossary · Phase 5 · Transformers

CLIP

OpenAI's 2021 model, trained on image–caption pairs to place matching pictures and text close together in one shared embedding space.

Try it now

An image model is a text model with one new first step. That's why it fits so neatly in a language model.

Multimodal transformers · How models see images and hear audio. About 35 minutes.

embedding space: The map that embeddings live on, where nearby points mean similar things. CLIP puts images and text on the same map.
patch embedding: The layer that turns each small square of an image into a list of numbers, like a token embedding for pictures.
projector: A small layer that maps a vision encoder's vectors into a language model's embedding space, so the model can read them like tokens.
spectrogram: A picture of sound: how much energy each frequency has over time.
vision encoder: The part of a multimodal model that turns an image into a list of vectors, often trained CLIP-style.
zero-shot classification: Sorting things into labels the model was never trained on, by comparing each item with a text description of each label.

Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.