Glossary · Phase 5 · Transformers
Spectrogram
A picture of sound: how much energy each frequency has over time.
Try it now
An image model is a text model with one new first step. That's why it fits so neatly in a language model.
Learn it properly
Multimodal transformers · How models see images and hear audio. About 35 minutes.
Other terms from the same lesson
- CLIP: OpenAI's 2021 model, trained on image–caption pairs to place matching pictures and text close together in one shared embedding space.
- embedding space: The map that embeddings live on, where nearby points mean similar things. CLIP puts images and text on the same map.
- patch embedding: The layer that turns each small square of an image into a list of numbers, like a token embedding for pictures.
- projector: A small layer that maps a vision encoder's vectors into a language model's embedding space, so the model can read them like tokens.
- vision encoder: The part of a multimodal model that turns an image into a list of vectors, often trained CLIP-style.
- zero-shot classification: Sorting things into labels the model was never trained on, by comparing each item with a text description of each label.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.