Pretraining at scale

Finding near-duplicates

Exact copies are easy to catch. You hash each document, which turns it into a short fingerprint code. Identical documents get identical fingerprints, so a quick lookup finds them.

Near-copies are harder. Two product pages might differ by a single word. To catch them, you need a score for how alike two documents are. A standard one works in two steps.

1. Break each document into overlapping three-word phrases, called shingles. "The tent is light" gives two shingles: "the tent is" and "tent is light".
2. Compute the Jaccard similarity. Count the shingles the two documents share. Divide by the number of different shingles they have between them.

Think of two friends' playlists. Count the songs on both lists, then divide by all the songs on either list. That share is how similar their taste is.

In plain words: Jaccard similarity is the share of phrases two documents have in common. A score of 1 means they have exactly the same phrases. A score of 0 means they share none.

Here it is on the practice catalog you trained your tiny GPT on. The code scores every pair of descriptions, then counts the pairs above three thresholds.

import re
import requests

docs = requests.get("https://api.gradient.test/v1/corpus").json()["text"].strip().split("\n")

def shingles(text, n=3):
    words = re.findall(r"[a-z0-9]+", text.lower())
    return {tuple(words[i:i + n]) for i in range(len(words) - n + 1)}

def jaccard(a, b):
    return len(a & b) / len(a | b)

sets = [shingles(d) for d in docs]
pairs = sorted(((jaccard(sets[i], sets[j]), i, j)
                for i in range(len(docs)) for j in range(i + 1, len(docs))), reverse=True)
print(f"{len(docs)} documents, {len(set(docs))} distinct as exact strings")
for threshold in (0.8, 0.6, 0.5):
    print(f"pairs sharing at least {threshold:.0%} of their phrases: {sum(p[0] >= threshold for p in pairs)}")
score, i, j = pairs[0]
print(f"\nclosest pair ({score:.2f}):\n  {docs[i]}\n  {docs[j]}")

No two descriptions are identical. Yet the closest pair differs only in the product's name. An exact-match check would keep both.

Comparing every pair works for a few hundred documents. At web scale it's impossible. Four documents make 6 pairs, but a billion documents make about $5 \times 10^{17}$ pairs. That's a 5 followed by 17 zeros. So pipelines use MinHash, a hashing trick. It estimates Jaccard similarity and puts likely duplicates in the same bucket, without comparing everything.

import re

def shingles(text, n=3):
    words = re.findall(r"[a-z0-9]+", text.lower())
    return {tuple(words[i:i + n]) for i in range(len(words) - n + 1)}

a = shingles("The tent is light and warm.")
b = shingles("The tent is light and dry.")
print(len(a), len(b))
print("shared:", len(a & b))
print("all different:", len(a | b))
print("Jaccard:", len(a & b) / len(a | b))

How much compute?

Training is mostly arithmetic. Engineers count it in FLOPs, short for floating-point operations. One FLOP is one multiply or one add on decimal numbers. A rule of thumb estimates the total from two numbers:

• $N$, the number of parameters (the model's adjustable weights);
• $D$, the number of training tokens the model reads.

In plain words: training costs about 6 FLOPs for every parameter, for every token. Double the model or double the data, and the cost doubles.

Where does the 6 come from? For each token, the forward pass (making a guess) costs about 2 FLOPs per parameter: one multiply and one add. The backward pass (working out how to fix each weight) costs about twice that, so 4 more. People write the rule as $6ND$ for short.

Try it with small numbers. A toy model with 1,000 parameters reads 50 tokens. That's 6 × 1,000 × 50 = 300,000 FLOPs. A laptop does that in well under a millisecond.

params = 1_000      # N: parameters
tokens = 50         # D: training tokens
flops = 6 * params * tokens
print(f"{flops:,} FLOPs")

To turn FLOPs into time, divide by how many FLOPs a chip does per second. An Nvidia H100 GPU peaks at about 989 trillion FLOPs per second with 16-bit numbers. Real training never reaches that peak. Chips spend much of their time waiting for data to move between memory and chips. The share of peak speed you actually get is called utilization. Large runs typically reach 30–50%, so the code below uses 40%.

def training_flops(params, tokens):
    return 6 * params * tokens

def readable(seconds):
    for unit, size in [("years", 365 * 86400), ("days", 86400), ("hours", 3600), ("minutes", 60)]:
        if seconds >= size:
            return f"{seconds / size:,.1f} {unit}"
    return "under a second" if seconds < 1 else f"{seconds:.0f} seconds"

runs = [
    ("our tiny GPT", 38_135, 300 * 16 * 24),       # 300 steps of 16 windows of 24 tokens
    ("GPT-3 (2020)", 175e9, 300e9),
    ("Chinchilla (2022)", 70e9, 1.4e12),
    ("Llama 3.1 405B (2024)", 405e9, 15.6e12),
]
per_gpu = 989e12 * 0.40      # an H100 at 40% of its peak 16-bit speed, in FLOPs per second
for name, n, d in runs:
    c = training_flops(n, d)
    print(f"{name:<22} {c:8.1e} FLOPs   one GPU: {readable(c / per_gpu):>14}   16,000 GPUs: {readable(c / per_gpu / 16_000)}")

Python shows huge numbers in e-notation. 2.6e+10 means 2.6 with the decimal point moved 10 places right: 26 billion. That's your tiny GPT, done in under a second. Llama 3.1 405B needs about 3.8e+25 FLOPs. On one GPU that would take over 3,000 years. On 16,000 GPUs it takes about 70 days.

Meta reported training Llama 3.1 405B on up to 16,000 H100 GPUs. Our estimate of about 70 days of pure compute is in line with its published figures. (GPT-3 originally trained on older V100 GPUs. The table uses today's hardware so you can compare.) Cloud GPUs rent for a few dollars per hour. So a run like that costs tens of millions of dollars. That's before counting failed experiments, staff and data.

params = 2e9
tokens = 1e12
print(6 * params * tokens)

Engineering at scale

At this size, the engineering matters as much as the model. Training is split across thousands of GPUs. Each GPU holds a slice of the model and a slice of each batch of data. Numbers are stored in 16 bits, or even 8, to save memory and time.

Hardware also fails, often. So the run saves checkpoints regularly. A checkpoint is a saved copy of the model, like a save point in a video game. After a crash, training restarts from the last one. Meta's Llama 3 report describes hundreds of interruptions during a single run.

What comes out: a base model

The result is a base model: a very good document completer. It works like your phone's autocomplete, at a far higher level. Give it the start of a product page, and it writes a believable rest of the page. Ask it to write a tagline and it doesn't obey: it continues the text as if it were a document, which may or may not contain a tagline.

It has picked up grammar, facts, coding patterns and reasoning habits from its data. But it doesn't follow instructions reliably, and it doesn't act as an assistant. Those skills come from post-training, in Phase 6.

Two practical results follow.

• Knowledge cutoff. The model knows the world only as it was when its data was collected. Anything later must come from the prompt, from tools or from retrieval.
• The data is the model. Gaps, biases and errors in the training text show up in the model. What the web says most often, the model says most confidently.

For marketers: if your brand, product type or market is rarely discussed online, a model knows little about it. That's one reason to ground answers in your own documents. This is called retrieval-augmented generation (RAG): the system looks up your documents and adds them to the prompt.