Quantization

So the choice that matters most is how many weights share a scale. The code below builds a 64 × 64 grid (a matrix) of small, typical weights: 4,096 in all. Then it plants one outlier, 50 times larger than usual. It quantizes the matrix to 4 bits in three ways:

• one scale for the whole matrix;
• one scale per row of 64 weights;
• one scale per group of 16 weights.

For each, it prints the average error on the ordinary weights, as a share of a typical weight's size.

import numpy as np

def quantize(w, bits, group):
    """Round-to-nearest with one scale per group of consecutive weights."""
    levels = 2 ** (bits - 1) - 1
    g = w.reshape(-1, group)
    scale = np.abs(g).max(axis=1, keepdims=True) / levels
    scale[scale == 0] = 1
    return (np.clip(np.round(g / scale), -levels, levels) * scale).reshape(w.shape)

rng = np.random.default_rng(0)
w = rng.normal(0, 0.02, (64, 64))                       # typical small weights...
w[3, 7] = 1.0                                           # ...and one outlier, 50x larger
for name, group in [("one scale for the matrix", 64 * 64), ("one scale per row", 64), ("one scale per 16 weights", 16)]:
    error = np.abs(quantize(w, 4, group) - w)
    error[3, 7] = 0                                     # judge the ordinary weights
    print(f"4-bit, {name:<26} average error {error.mean() / np.abs(w).mean():.0%} of a typical weight")

With one scale for the whole matrix, the error is 98% of a typical weight. The ordinary weights lose almost everything. One scale per row cuts the error to 13%, because only the outlier's own row is badly hit. Groups of 16 bring it down to 9%, since each small group's scale fits its own values more closely.

In practice, groups of 32, 64 or 128 weights are common. Smaller groups are more accurate. But every group stores its own scale, and those scales take space too.

weights = [0.7, -0.32, 0.18, 0.04, -0.51, 0.26]
for bits in [8, 4, 3, 2]:
    levels = 2 ** (bits - 1) - 1
    scale = max(abs(w) for w in weights) / levels
    rebuilt = [round(w / scale) * scale for w in weights]
    error = sum(abs(r - w) for r, w in zip(rebuilt, weights)) / len(weights)
    print(f"{bits}-bit: levels -{levels} to {levels}, rebuilt {[round(r, 2) for r in rebuilt]}, average error {error:.3f}")

A real model at each precision

Now try it on a real model: the course's tiny GPTs. The code quantizes every weight matrix in groups of 16, at 8, 4, 3 and 2 bits. It compares each with the original 32-bit model and prints three things:

• Size in kilobytes (KB), counting the 16-bit scales too.
• Quality: the base model's loss on catalog text. Lower is better.
• Writing: the start of the chat model's answer to a product question.

(torch here is the course's in-browser PyTorch subset.)

import numpy as np
import requests
import torch
import torch.nn.functional as F
from tinygpt import load_pretrained

def quantize(w, bits, group=16):
    levels = 2 ** (bits - 1) - 1
    g = w.reshape(-1, group)
    scale = np.abs(g).max(axis=1, keepdims=True) / levels
    scale[scale == 0] = 1
    return (np.clip(np.round(g / scale), -levels, levels) * scale).reshape(w.shape)

def quantized(weights, bits):
    return {k: torch.tensor(quantize(v.numpy(), bits)) if v.ndim == 2 else v for k, v in weights.items()}

base, tok = load_pretrained("tiny-gpt")
chat, _ = load_pretrained("tiny-gpt-chat")
base_w, chat_w = base.state_dict(), chat.state_dict()
data = torch.tensor(tok.encode(requests.get("https://api.gradient.test/v1/corpus").json()["text"]))
torch.manual_seed(0)
starts = torch.randint(0, len(data) - 49, (64,)).tolist()
x = torch.stack([data[i:i + 48] for i in starts])
y = torch.stack([data[i + 1:i + 49] for i in starts])
n = sum(v.numel() for v in base_w.values())
prompt = torch.tensor([tok.encode("<user> Describe the Cove camp mug . <assistant>")])

for bits in (32, 8, 4, 3, 2):
    if bits < 32:
        base.load_state_dict(quantized(base_w, bits))
        chat.load_state_dict(quantized(chat_w, bits))
    with torch.no_grad():
        loss = F.cross_entropy(base(x).reshape(-1, len(tok)), y.reshape(-1)).item()
    torch.manual_seed(0)
    reply = tok.decode(chat.generate(prompt, 40, temperature=0.7, stop_id=tok.token_id("<end>"))[0, prompt.shape[1]:].tolist())
    size_kb = (n * bits / 8 + (n / 16 * 2 if bits < 32 else 0)) / 1024
    print(f"{bits:>2}-bit  {size_kb:6.1f} KB  loss {loss:.3f}  | {reply[:95]}")

The pattern is typical of real models.

• 8-bit is essentially free. The loss is unchanged, at a little over a quarter of the 32-bit size.
• 4-bit costs a little. The loss rises slightly, and the writing is still fine.
• 3-bit starts to hurt.
• 2-bit breaks the model. It writes a jumble of catalog words.

Larger models handle quantization better than this tiny one, because each weight matters less.

Watch out: the scales take space. Each group of 16 weights also stores one 16-bit scale. That's 16 ÷ 16 = 1 extra bit per weight, so "4-bit" here really costs 5 bits per weight. That's one reason real formats use larger groups.

bits, group, scale_bits = 4, 32, 16
print(bits + scale_bits / group)

What it buys

The code below works out the memory and a top speed for two real open models. Llama 3 8B has 8 billion weights, and Llama 3 70B has 70.6 billion. Each is shown at three precisions.

models = {"Llama 3 8B": 8.0e9, "Llama 3 70B": 70.6e9}
formats = {"16-bit": 16, "8-bit": 8, "4-bit (about 4.5 bits with scales)": 4.5}
for name, params in models.items():
    print(name)
    for fmt, bits in formats.items():
        gb = params * bits / 8 / 1e9
        print(f"  {fmt:<36} {gb:6.1f} GB of weights   decode limit at 1 TB/s: ~{1e12 / (params * bits / 8):.0f} tokens/s")

Both gains come from moving fewer bytes.

• Memory. The 70B model drops from about 141 GB to about 40 GB. That takes it from two or more datacenter GPUs down to one. The 8B model drops to about 4.5 GB, which fits comfortably on a laptop. Freed memory can also hold more KV cache, so a server can handle more users at once.
• Speed. As Phase 5 showed, producing each token means reading every weight from memory. So the speed of writing text is limited by the bytes read per token. Half the bytes gives roughly twice the tokens per second: about 62, then 125, then 222 for the 8B model. These are rough upper limits. They assume memory that moves 1 TB (a trillion bytes) per second, and they ignore everything else.

Things to know

• Formats. You'll meet round-to-nearest int8 (8-bit integers) and GPTQ and AWQ (4-bit, for GPUs). There's also GGUF, the file format of llama.cpp and Ollama. Its quantization levels have names like Q4_K_M and Q8_0. Recent GPUs also compute directly with FP8 and FP4, small floating-point formats. Some labs now train in 8-bit. Some also ship "quantization-aware" checkpoints: saved models trained to survive 4-bit.
• What gets quantized. Usually the big weight matrices. Embeddings, normalization layers and sometimes the output layer stay in higher precision. The KV cache can be quantized too.
• Evaluate after quantizing. An average loss can hide damage to specific skills, such as math, rare languages or long documents. Before you deploy a 3- or 4-bit model, compare it on your own task.

For marketers: when a provider offers a cheaper "quantized" or "turbo" version of a model, treat it as a different model. Run your own evaluation set on it before you switch high-stakes work.