Chat template: what it is, with an example

The fixed format, with special tokens, that marks who is speaking in a conversation, so a chat model knows where your message ends and its reply starts.

What it means

A language model reads one long sequence of tokens, but a conversation is a list of messages. A chat template is the fixed format that turns one into the other. Special tokens mark who's speaking and where each message ends, so the model can tell your message from its own reply. The last piece is a cue that says it's the assistant's turn, and the model carries on from there.

An example in Python

messages = [
    {"role": "system", "content": "You write product copy."},
    {"role": "user", "content": "Describe the Lark tent."},
]

def render(messages):
    parts = [f"<|im_start|>{m['role']}\n{m['content']}<|im_end|>" for m in messages]
    parts.append("<|im_start|>assistant\n")      # the model carries on from here
    return "\n".join(parts)

print(render(messages))

What it prints

<|im_start|>system
You write product copy.<|im_end|>
<|im_start|>user
Describe the Lark tent.<|im_end|>
<|im_start|>assistant

This is the ChatML style, which several models use. Others use different tokens, and that's what matters: a model only understands the format it was trained on. A wrong template is a common reason an open model rambles, never stops, or answers its own question.

In practice

Hosted APIs apply the template for you, which is why you can send a list of messages. With open models the template ships with the tokenizer, and in Hugging Face Transformers you call tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True). The last argument adds the assistant cue. Forgetting it is an easy mistake to make.

Chat template

What it means

An example in Python

In practice

Try it now

Learn it properly

Other terms from the same lesson

Never trained an AI?