Conversations and memory

What to store for the assistant's turn

The helper above stores the reply's text. That's all a plain conversation needs, and it's what you'll do in this lesson.

Long conversations get slower and cost more

Think back to the friend who needs the whole thread pasted in. Every new message makes the paste longer. Requests work the same way. Each one resends the whole conversation as input tokens. (A token is a word or a piece of a word, and you pay for every one you send.)

Here's the size of one request, in words:

In plain words: tokens in this request = tokens per earlier turn × number of earlier turns + tokens in the new message.

Work out turn 3 with small numbers. Say each turn (one user message plus one reply) is about 300 tokens. The new message is about 100 tokens. Turn 3 has two earlier turns:

• earlier turns: 300 × 2 = 600 tokens
• new message: 100 tokens
• this request: 600 + 100 = 700 tokens

Turn 1 sent only 100 tokens. Turn 3 sends 700. The code below runs the same sum for 20 turns and keeps a running total:

tokens_per_turn = 300   # one user message plus one reply, roughly
total = 0
for turn in range(1, 21):
    this_request = tokens_per_turn * (turn - 1) + 100   # all earlier turns, plus the new message
    total += this_request
    if turn in (1, 5, 10, 20):
        print(f"Turn {turn}: {this_request} input tokens in this request, {total} sent so far")

By turn 20, one request is 5,800 tokens. Twenty short turns send 59,000 input tokens in total. Yet the conversation itself is only about 6,000 tokens long. Each turn costs more than the last, so the total climbs much faster than the number of turns.

Long histories also take longer to process. A very long one can eventually pass the model's context window, the most text it can handle in one request.

Keeping history in check

You have four common options. Each trades cost against what the model gets to see.

• Keep the last N messages. N is a number you pick, such as 10. This is called a sliding window, like a group chat that only shows the latest messages. It's simple and cheap, but early details vanish. Examples: the product the user named in their first message, or a decision made an hour ago.
• Summarize older turns. Ask the model for a short summary of the early conversation, like meeting notes. Then send that summary plus the recent turns. Key facts survive. The cost is an extra call and some lost detail.
• Pin the facts that matter. Keep a short list of decisions and details in code: the campaign, the audience, the agreed tone. Put it in the system prompt on every request.
• Let the API do it. Anthropic's API offers server-side compaction. It summarizes older context automatically once a conversation gets long. It was a beta feature at the time of writing.

def trim_history(history, max_messages):
    trimmed = history[-max_messages:]
    if trimmed and trimmed[0]["role"] == "assistant":
        trimmed = trimmed[1:]
    return trimmed

history = [
    {"role": "user", "content": "Write a subject line for the rain jacket sale."},
    {"role": "assistant", "content": "Rain jackets 30% off, through Sunday"},
    {"role": "user", "content": "Make it shorter."},
    {"role": "assistant", "content": "30% off rain jackets"},
    {"role": "user", "content": "Now one for the fleece launch."},
    {"role": "assistant", "content": "Meet the Ridgeline Fleece"},
    {"role": "user", "content": "Add the price."},
]

recent = trim_history(history, 4)
print(len(history), "messages in the full history,", len(recent), "sent")
for message in recent:
    print(message["role"], "|", message["content"])

Memory is a separate thing

Think of your phone. The thread with a friend is one thing. The note you saved on their contact card ("allergic to nuts") is another. The note is still there when you start a new thread.

Apps work the same way:

• Conversation history is the transcript of one chat.
• Long-term memory is facts saved outside any single chat, in a database or a file. The app adds them to future requests.

When an app "remembers" that you like short drafts, it stored a note. Later, it put that note in the system prompt of a new conversation. The model itself learned nothing.

You help a marketing team write email copy for Trailhead Outfitters.

<user_notes>
- Prefers short drafts, no emoji.
- Works on the email newsletter, not paid social.
</user_notes>

Memory is data you keep about a person. Design it like any other customer data:

• Store what's useful, for a clear purpose.
• Tell users what's remembered.
• Let them edit or delete it.
• Never store secrets such as passwords or card numbers.

Business data isn't memory either. Budgets, order statuses, stock levels and campaign results live in your real systems. They can change at any time. Say a user mentioned "a $5,000 budget" thirty turns ago, and finance has since cut it. The conversation is now out of date. When you need the figure, fetch it from the system that owns it. Put that current figure in the request.