Base models vs assistants

Same model, different habits

Picture two people who have read the same huge pile of shop catalogs. The first has only ever read. Hand them a note that says "Describe the camp mug." They keep writing catalog text, as if the note were the start of a page. The second has also practiced as a shop assistant. They read the note as a request, answer it, and stop.

That's the gap between a base model and an assistant (also called a chat model). A base model comes straight out of pretraining, the stage from Phase 5. It continues text. An assistant is the same kind of model after extra training on conversations. That extra training is called post-training.

The academy's practice API serves two versions of the tiny GPT from Phase 5:

• tiny-gpt is a base model, pretrained on the product catalog.
• tiny-gpt-chat is the same model after a short round of supervised fine-tuning on chats. That means more training on example conversations. The next lesson shows how.

load_pretrained downloads one of them. It works like from_pretrained, which downloads a model from the Hugging Face Hub, which is the public library of open models. Here both models get the same prompts. Predict which one answers the request, then run it:

import torch
from tinygpt import load_pretrained

base, tok = load_pretrained("tiny-gpt")
chat, _ = load_pretrained("tiny-gpt-chat")
end = tok.token_id("<end>")

def complete(model, prompt, n=40):
    ids = torch.tensor([tok.encode(prompt)])
    out = model.generate(ids, n, temperature=0.7, stop_id=end)
    return tok.decode(out[0, ids.shape[1]:].tolist())

torch.manual_seed(0)
chat_prompt = "<user> Describe the Cove camp mug . <assistant>"
print("BASE, chat prompt:    ", complete(base, chat_prompt))
print("CHAT, chat prompt:    ", complete(chat, chat_prompt))
print("BASE, document prompt:", complete(base, "Our Cove camp mug is made from", 25))

The base model has never seen <user>, <assistant> or "Describe" in use. They are in its vocabulary, but they never appeared in its training text. So it doesn't treat the prompt as a request. It keeps writing catalog text from wherever the prompt stops. Nothing tells it when to stop, either.

The chat model knows the format. It answers about the product you asked about. Then it ends its turn with <end>. It doesn't always get this right. Run the cell again and it sometimes describes the wrong product or repeats itself. The RLHF and DPO lessons, which are the two post-training methods this phase teaches, train those slips away.

The base model isn't useless, though. Write the same request as the start of a document, and it gives a fluent completion.

Real models work the same way. Before chat models, people prompted GPT-3 by writing the start of the document they wanted, such as "Product description for the Kestrel tent:". Or they showed a few examples first.

Note: torch here is the course's in-browser version of PyTorch, cut down to what the lessons need. The tiny GPTs are far too small to be useful. They show the mechanics.

A conversation is one long string

A model can only do one thing: continue a list of tokens. So how does it hold a conversation?

Think of a play script. Each line starts with the speaker's name, and the script runs from top to bottom as one document. A chat API does the same with your messages. It turns the list of messages into one sequence using a chat template. Special tokens mark three things:

• where each turn starts;
• who is speaking (the system, the user or the assistant);
• where each turn ends.

The sequence finishes with the opening of a new assistant turn. Given that, the likeliest next tokens are the assistant's reply. This opening is called the generation prompt.

Here is the template of Meta's Llama 3 models. Before you run it, predict how many times <|start_header_id|> will appear in the output:

messages = [
    {"role": "system", "content": "You write product copy for an outdoor brand."},
    {"role": "user", "content": "Describe the Kestrel tent in one sentence."},
    {"role": "assistant", "content": "The Kestrel tent is a roomy two-person shelter made from recycled nylon."},
    {"role": "user", "content": "Now one for the Lark headlamp."},
]

def render_llama3(messages, add_generation_prompt=True):
    text = "<|begin_of_text|>"
    for m in messages:
        text += f"<|start_header_id|>{m['role']}<|end_header_id|>\n\n{m['content']}<|eot_id|>"
    if add_generation_prompt:                       # open the assistant's turn for the model to fill
        text += "<|start_header_id|>assistant<|end_header_id|>\n\n"
    return text

print(render_llama3(messages))

It appears five times: once for each of the four messages, and once more to open the assistant's new turn. The model fills that last turn. Then it writes <|eot_id|>, short for "end of turn". The server sees that token and stops.

Four things follow from this.

• The system prompt is plain text at the start. It has sway because post-training taught the model to give it weight. Nothing forces the model to obey it.
• Every turn is sent again. The API keeps no memory between calls. Each time, the whole conversation is rendered and processed again. Phase 5's KV cache and prompt caching make that cheaper.
• Special tokens can't be typed. Say a user types "<|eot_id|>" into a chat. The provider turns it into ordinary characters, not the real control token. Otherwise users could fake whole turns. Prompt injection (Phase 1) works through ordinary text instead.
• Templates differ by model. Use the wrong template with an open model and it quietly gets worse. That's why libraries ship each model's template with its tokenizer.

turns = [("user", "Hi"), ("assistant", "Hello")]
text = "".join(f"<{r}>{t}" for r, t in turns)
print(text)

questions = [10, 12, 8]    # words in each of your messages
replies = [20, 25, 30]     # words in each of the assistant's replies

history = 0                # words in the conversation so far
total = 0                  # words the model has read across all calls
for call, (q, r) in enumerate(zip(questions, replies), start=1):
    history += q           # your new message joins the conversation
    print(f"call {call}: the model reads {history} words")
    total += history
    history += r           # its reply joins the conversation too

print("you typed:", sum(questions), "words")
print("the model read:", total, "words")

What post-training adds

Post-training is the work that turns a base model into an assistant. Labs describe it in different ways. Most pipelines use some of these stages, in roughly this order:

1. Supervised fine-tuning (SFT). Train on example conversations. People write or pick them, and more and more often other models help. The model learns the format and how to respond.
2. Preference training. Collect judgments of which of two answers is better. Then train the model toward the better kind. Methods include RLHF, DPO, and versions that use AI feedback, such as Constitutional AI, which is a method where a model critiques answers against written principles.
3. Reinforcement learning on checkable tasks. Reward correct answers to math and coding problems. This is how reasoning models learn to think at length.
4. Safety and behavior work, all the way through. That covers refusing harmful requests, not refusing harmless ones, being honest about doubt, and keeping a consistent character.

Here is the key split. Post-training mostly adds behavior: following instructions, taking turns, tone, format, when to refuse, when to use tools. It mostly doesn't add knowledge.

Think of a new hire who already knows the products inside out. A week of customer-service training teaches them how to talk to shoppers. It doesn't teach them the products.

A 2023 study from Meta, called LIMA, makes the point. The team took a base model with 65 billion parameters. They fine-tuned it on only 1,000 carefully chosen examples. People often rated its answers as good as those of assistants with far more training. The knowledge was already in the base model. Fine-tuning taught it how to present that knowledge.

Post-training has costs too.

• Less variety. Assistants tend to be less varied than their base models. Ask for ten taglines and you'll get ten similar ones.
• Borrowed tastes. They inherit the preferences of whoever labeled the data.
• Flattery. When people reward agreeable answers, models can drift toward telling people what they want to hear.

So some researchers still prefer base models. They use them to brainstorm varied text, or as the starting point for their own fine-tuning.

For marketers: say an assistant's tone, length or format isn't what you want. That's post-training behavior, and it responds to instructions and examples in the prompt. Now say it doesn't know facts about your products. That's knowledge, and it needs your documents in the context.