Encoders and embedding models

A new training game: fill in the blanks

A decoder learns by guessing the next token. That game doesn't work for an encoder. It already sees the whole sentence, so the next token is sitting right there in its input. It could copy the answer and learn nothing.

So encoders play a different game. Think of a fill-in-the-blanks quiz at school. Some words are blanked out, and you use the words around each gap to fill it in. BERT, an encoder released by Google in 2018, introduced this for transformers. The method is called masked language modeling. About 15% of the tokens are chosen and hidden, usually behind a special [MASK] token. The model must guess each hidden word from the context on both sides.

The code below picks the blanks for one product sentence.

import re
import torch

sentence = "The Cedar rain jacket is made from recycled nylon . It is light and fully waterproof ."
tokens = sentence.split()
torch.manual_seed(3)
masked = torch.rand(len(tokens)) < 0.15
inputs = [("[MASK]" if m else t) for t, m in zip(tokens, masked.tolist())]
targets = [t for t, m in zip(tokens, masked.tolist()) if m]
print("model sees:   ", " ".join(inputs))
print("must predict: ", targets)
print(f"training signals from this sentence: {len(targets)} of {len(tokens)} tokens")

Here's what each piece does.

• sentence.split() cuts the sentence at every space, giving 17 tokens. Real models use a tokenizer, as in the tokenization lesson. Splitting at spaces keeps the example easy to read.
• torch.rand(len(tokens)) draws a random number between 0 and 1 for each token. A token is hidden when its number is below 0.15. So each token has a 15% chance.
• inputs is what the model sees, with [MASK] in place of each hidden token. targets lists the hidden words it must guess.

Each token is picked by chance, so one short sentence can get more or fewer than 15% hidden. Over millions of sentences, the share settles very close to 15%.

Here's the catch. Only the hidden positions give a training signal: a guess that gets graded. That's about 15% of the tokens. A GPT-style decoder makes a graded guess at every position. So masked training learns less from each sentence. That's one reason decoder-style pretraining won out for the largest models. (Pretraining is the first, longest stage of training, on huge amounts of text.)

Turning a whole text into one vector

In Phase 1 you used embeddings. An embedding is a list of numbers that captures what a text means, so you can compare texts by meaning. Most embedding models are encoders underneath.

An encoder gives back one vector per token. A sentence of 12 tokens gives 12 vectors. Search needs one vector per text, so the model squashes the token vectors into one. This step is called pooling. The most common way is mean pooling: take the average. Add up the vectors number by number, then divide by how many there are.

Think of a team sheet. Each player has a score for speed and a score for strength. The team average gives one speed number and one strength number for the whole team. Mean pooling does the same with token vectors.

A plain encoder doesn't make a good embedding model straight away. It's trained further with contrastive learning. The model sees pairs that belong together, such as a question and its answer. Training pulls their vectors closer. It pushes the vectors of unrelated texts apart. The result is an embedding model like the ones used for semantic search and RAG.

In plain words: an embedding model is an encoder plus an averaging step. It's trained so that texts which mean the same thing end up with similar vectors.

vecs = [[2.0, 4.0], [4.0, 0.0]]
print([(a + b) / 2 for a, b in zip(*vecs)])

tokens = [[1.0, 2.0], [3.0, 4.0], [5.0, 0.0], [0.0, 0.0]]   # the last row is padding
keep = [True, True, True, False]
kept = [vec for vec, k in zip(tokens, keep) if k]
mean = [sum(vec[d] for vec in kept) / len(kept) for d in range(2)]
wrong = [sum(vec[d] for vec in tokens) / len(tokens) for d in range(2)]
print("mean of the real tokens:", mean)
print("mean with padding:      ", wrong)

Classifiers and rerankers

Classifiers. A classifier sorts texts into labels you choose. Is a review positive or negative? Which topic is this email about? Is this comment abusive? To build one, add a small layer on top of the pooled vector that picks a label. Then fine-tune: train the whole model a little more on your own labeled examples. Small encoders do this fast and cheaply. That matters when you have millions of texts.

Rerankers. You met reranking in Phase 1. A search first fetches candidates by comparing embeddings. Then a reranker re-scores the best few. Most rerankers are cross-encoders. A cross-encoder reads the query and one document together, as a single input. It outputs one relevance score.

Why is that more accurate? With separate embeddings, the query and the document never meet. Each is squashed into a vector on its own, and then the two vectors are compared. In a cross-encoder, every query token can attend to every document token. It can notice that "waterproof" in the query matches "fully waterproof" in a product page.

The cost is speed. Document embeddings can be worked out once, ahead of time, and stored. A cross-encoder has to run again for every pair of query and document. So pipelines fetch candidates with embeddings and rerank only the top few with a cross-encoder.

Encoder-decoder models

The original transformer, from 2017, had both halves. Picture an interpreter. First they hear the whole sentence. Then they speak the translation, glancing back at what they heard.

The encoder reads the source sentence, with attention in both directions. The decoder writes the translation one token at a time, with a causal mask. To look back at the source, the decoder uses cross-attention. The queries come from the decoder. The keys and values come from the encoder's outputs. So each word being written can look up the parts of the source it needs.

T5 and BART are well-known encoder-decoder models. Today most general assistants are decoder-only. Encoder-decoders are still common in translation and speech recognition.

Choosing a model

For marketers: say you tag a million product reviews by theme every night. That's a job for a small fine-tuned encoder, or for embeddings plus clustering (grouping similar vectors). It's not a job for a chat model at a fraction of a cent per call, a million times a night.