Glossary · Phase 5 · Transformers
Embedding space
The map that embeddings live on, where nearby points mean similar things. CLIP puts images and text on the same map.
After training, images and text live in one shared embedding space. A picture and its caption land close together. The test score climbs from chance, about 10% with 10 concepts, to over 90%.
Try it now
An image model is a text model with one new first step. That's why it fits so neatly in a language model.
Learn it properly
Multimodal transformers · How models see images and hear audio. About 35 minutes.
Other terms from the same lesson
- CLIP: OpenAI's 2021 model, trained on image–caption pairs to place matching pictures and text close together in one shared embedding space.
- patch embedding: The layer that turns each small square of an image into a list of numbers, like a token embedding for pictures.
- projector: A small layer that maps a vision encoder's vectors into a language model's embedding space, so the model can read them like tokens.
- spectrogram: A picture of sound: how much energy each frequency has over time.
- vision encoder: The part of a multimodal model that turns an image into a list of vectors, often trained CLIP-style.
- zero-shot classification: Sorting things into labels the model was never trained on, by comparing each item with a text description of each label.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.