Index Ai Academy
Glossary · Phase 5 · Transformers

Attention mask

A grid of allowed and blocked positions that sets which tokens each token may attend to.

Try it now

Press Next. Rows are the token reading, columns the token being read. Struck cells are blocked.

Learn it properly

Encoders and embedding models · BERT-style models and where they fit. About 40 minutes.

Other terms from the same lesson

  • BERT: Google's 2018 encoder model, trained by hiding words and guessing them from both sides. Its descendants still power much search and classification.
  • bidirectional: Reading in both directions: each token can look at the words before and after it, as encoders like BERT do.
  • contrastive learning: Training that pulls the embeddings of matching pairs (like a question and its answer) together and pushes mismatched pairs apart.
  • cross-attention: Attention where one sequence looks at another, such as a translation model's decoder reading the encoder's output.
  • cross-encoder: A model that reads a query and a document together and gives one relevance score. More accurate than comparing embeddings but slower, so it reranks a short list.
  • encoder: A transformer that reads the whole text at once to understand it, used for embeddings and classification.
  • masked language modeling: Training by hiding some tokens and asking the model to guess them from the words on both sides. It's how BERT learned.
  • mean pooling: Turning a text's token vectors into one vector for the whole text by averaging them.
  • padding mask: A mask that stops attention from looking at padding tokens added to make sequences the same length.
Keep learning

More in plain English.

1286 terms in the glossary, each with the lesson that teaches it.

1286 terms

The glossary

Every word you'll meet in AI and machine learning, explained in everyday language.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write