Contrastive learning

Training that pulls the embeddings of matching pairs (like a question and its answer) together and pushes mismatched pairs apart.

A plain encoder doesn't make a good embedding model straight away. It's trained further with contrastive learning. The model sees pairs that belong together, such as a question and its answer. Training pulls their vectors closer. It pushes the vectors of unrelated texts apart. The result is an embedding model like the ones used for semantic search and RAG.

Other terms from the same lesson

• BERT: Google's 2018 encoder model, trained by hiding words and guessing them from both sides. Its descendants still power much search and classification.
• bidirectional: Reading in both directions: each token can look at the words before and after it, as encoders like BERT do.
• cross-attention: Attention where one sequence looks at another, such as a translation model's decoder reading the encoder's output.
• cross-encoder: A model that reads a query and a document together and gives one relevance score. More accurate than comparing embeddings but slower, so it reranks a short list.
• encoder: A transformer that reads the whole text at once to understand it, used for embeddings and classification.
• masked language modeling: Training by hiding some tokens and asking the model to guess them from the words on both sides. It's how BERT learned.
• mean pooling: Turning a text's token vectors into one vector for the whole text by averaging them.