BERT

Google's 2018 encoder model, trained by hiding words and guessing them from both sides. Its descendants still power much search and classification.

Other terms from the same lesson

• bidirectional: Reading in both directions: each token can look at the words before and after it, as encoders like BERT do.
• contrastive learning: Training that pulls the embeddings of matching pairs (like a question and its answer) together and pushes mismatched pairs apart.
• cross-attention: Attention where one sequence looks at another, such as a translation model's decoder reading the encoder's output.
• cross-encoder: A model that reads a query and a document together and gives one relevance score. More accurate than comparing embeddings but slower, so it reranks a short list.
• encoder: A transformer that reads the whole text at once to understand it, used for embeddings and classification.
• masked language modeling: Training by hiding some tokens and asking the model to guess them from the words on both sides. It's how BERT learned.
• mean pooling: Turning a text's token vectors into one vector for the whole text by averaging them.