Running open models locally

How fast will it run?

To write each new token (a word or piece of a word), a typical model reads every one of its weights from memory. Picture a cook who must read the whole recipe book before adding each ingredient. A thicker book means slower cooking.

So speed depends on memory bandwidth: how many gigabytes the chip can read from memory each second.

In plain words: divide how fast the machine reads memory by how much it has to read for each token. That gives the most tokens per second you can hope for.

Take the same 4.8 GB model on a laptop that reads 100 GB per second. 100 ÷ 4.8 is about 21. So the model writes at most about 21 tokens per second.

weights_gb = 4.8      # the 8B model at Q4_K_M
bandwidth = 100       # GB the laptop reads per second
print(round(bandwidth / weights_gb))

It prints 21. This is an upper limit. Real speeds are usually 60–80% of it.

print(round(400 / (27e9 * 4.8 / 8 / 1e9)))

Four machines, three models

Now put both sums together. The code below checks three model sizes on four real machines. Two are laptops. The other two are GPUs, chips built to do many sums at once, which makes them good at running models. Each model is tried in three formats: full 16-bit, Q8_0 and Q4_K_M. For every combination that fits, it prints the top speed in tokens per second ("tok/s").

A few things to know before you run it:

• "Memory the model can use" is less than the machine's total. The operating system and other apps need the rest. On a 16 GB laptop the code assumes about 10 GB is free for the model.
• The KV cache figures are rough guesses for about 8,000 tokens of conversation. Bigger models need more.
• The bits per weight include the scales: small extra numbers that quantized formats store next to the weights (from the quantization lesson).

models = {"8B": (8e9, 1.0), "27B": (27e9, 2.0), "70B": (70e9, 2.5)}   # parameters, rough KV cache (GB) for 8K tokens
formats = {"16-bit": 16, "Q8_0": 8.5, "Q4_K_M": 4.8}                  # bits per weight, including scales
machines = {                                                          # memory the model can use (GB), bandwidth (GB/s)
    "16 GB MacBook Air (M3)": (10, 100),
    "36 GB MacBook Pro (M3 Max)": (27, 400),
    "24 GB gaming GPU (RTX 4090)": (22, 1008),
    "80 GB datacenter GPU (H100)": (78, 3350),
}
for machine, (memory, bandwidth) in machines.items():
    print(machine)
    for name, (params, kv_gb) in models.items():
        options = []
        for fmt, bits in formats.items():
            weights_gb = params * bits / 8 / 1e9
            if weights_gb + kv_gb + 1.0 <= memory:            # + ~1 GB for the runtime
                options.append(f"{fmt} ~{bandwidth / weights_gb:.0f} tok/s")
        print(f"  {name:>4}: {', '.join(options) if options else 'does not fit'}")

The speeds are upper limits, but the overall picture is right.

• An 8B model at 4-bit runs on an ordinary laptop. On the 16 GB MacBook Air it's the only option that fits, at up to about 21 tokens per second.
• A 27B model needs more. It fits at Q4_K_M on the high-memory laptop or the gaming GPU, at about 25 and 62 tokens per second.
• A 70B model needs a lot more. Of these four, only the datacenter GPU fits it. Otherwise you need two consumer GPUs or a Mac with 64 GB or more.

Apple Silicon Macs are popular for this. Their GPU (the chip that runs the model) shares the machine's large main memory, called unified memory. A gaming GPU has its own, smaller memory, but reads it much faster.

Tip: a rule of thumb from practice. For a fixed amount of memory, a bigger model at 4-bit usually beats a smaller model at 16-bit. Still test both on your own task.

memory, bandwidth, kv_gb = 27, 400, 1.5
for name, params, bits in [("14B Q8_0", 14e9, 8.5), ("14B Q4_K_M", 14e9, 4.8), ("32B Q8_0", 32e9, 8.5), ("32B Q4_K_M", 32e9, 4.8)]:
    weights_gb = params * bits / 8 / 1e9
    total = weights_gb + kv_gb + 1.0
    if total <= memory:
        print(f"{name:<11} {total:5.1f} GB of {memory}: fits, up to ~{bandwidth / weights_gb:.0f} tokens/s")
    else:
        print(f"{name:<11} {total:5.1f} GB of {memory}: does not fit")

Running one

Ollama is the easiest way to start. It's a free program that downloads quantized models, runs them, and answers requests from other programs on your computer. Other popular options are:

• LM Studio, a desktop app with a chat window;
• llama.cpp, the engine underneath many of these tools;
• MLX, Apple's framework for Apple Silicon Macs;
• vLLM, for serving models on GPUs to many users.

In a terminal on your own machine, two commands get you chatting:

ollama pull qwen2.5:7b          # downloads a 4-bit build, about 4.7 GB
ollama run qwen2.5:7b           # chat in the terminal

The first downloads the model. The second starts a chat. Ollama also runs a small local server: a program that waits for requests and sends back replies. Your Python code can talk to it. This code can't run inside the academy, because the academy can only reach its practice API.

import requests

reply = requests.post("http://localhost:11434/api/chat", json={
    "model": "qwen2.5:7b",
    "messages": [
        {"role": "system", "content": "You write product copy for an outdoor brand."},
        {"role": "user", "content": "Write a one-sentence description of a lightweight two-person tent."},
    ],
    "stream": False,
}).json()
print(reply["message"]["content"])

Here is what each part does:

• localhost means "this computer". 11434 is the port, a numbered door Ollama listens on. So the request never leaves your machine.
• messages is the conversation: a system message that sets the role, then the user's request.
• "stream": False asks for the whole reply at once, instead of word by word.

Ollama also offers a second web address, called an endpoint, that copies OpenAI's format. So many existing tools can use a local model if you change one web address. You don't need to format the chat template yourself. Ollama reads it from the model's files and applies it for you.

Local or hosted?

A hosted API is a model that runs on a provider's computers. You send text over the internet and pay per token. Here's how the two choices compare:

Frontier models are the newest, most capable models from the big labs. There are also middle grounds. You can rent GPUs in the cloud and run an open model on them. Or you can use providers that host open models behind an API.

Whichever you choose, test it on your own task. A model that tops a leaderboard (next lesson) may not be the best at writing your product copy.

For marketers: a local 7–8B model suits private, high-volume, simple jobs. Examples: tagging customer feedback that can't be sent to a third party, first drafts of routine copy, and pulling fields out of documents. For nuanced writing and analysis, a frontier hosted model is usually still worth its price.