Logging and monitoring
See what your LLM app actually did in production.
Create a free account Try part of it here
Why it matters
When a normal web page breaks, it shows an error. When an AI feature goes wrong, it usually returns a fluent, wrong answer with no error at all. Nobody notices until a customer posts a screenshot. Logging what every call received, returned, cost and how long it took is how you find those failures, explain them, and feed them back into your evals. It's also how you answer "why did the AI bill double this week?"
What you'll be able to do
- List what to record for every model call and every multi-step run
- Wrap model calls so they're timed and logged even when they fail
- Redact personal data before it reaches your logs
- Turn logs into the handful of numbers worth watching: cost, latency percentiles, errors, truncations
What it covers
- What to record for each call
- Keep personal data out
- Why the average hides the slow tail
- Quiet logs don't mean good answers
What to record for each call
Think of a plane's flight recorder. Nobody reads it on a normal day. When something goes wrong, it's the only way to find out what happened.
Here's what to record for every model call:
| Field | Why you want it |
|---|---|
| Time, feature name, prompt version | Which part of the product, and which version of the prompt, produced this |
| Model and effort | Explains differences in quality and cost |
The reply's id (and the request-id header) | Lets you look up a call with the provider's support team |
| Input and output (or a link to them) | You can't debug what you didn't keep; remove personal data first |
usage: input, output, cache read and write tokens | Cost, and whether caching works |
| Latency: time to first token and total | Users feel both |
stop_reason | max_tokens means cut-off answers; a jump in refusal needs a look |
| Tool calls and their results | What an agent actually did |
| Errors | Status code and message |
| User feedback | Thumbs up or down, edits, "regenerate" clicks: the cheapest quality signal you'll get |
Traces for multi-step work
An agent run or a prompt chain is many calls. A trace groups them under one id. Each step inside it, such as a model call, a tool call or a search, is a span with its own timing. When an agent gives a strange answer, the trace shows the exact path: which documents it found, which tool it called with what input, and where the time went.
For a small app, writing your own JSON log lines is enough. Purpose-built tools such as Langfuse, LangSmith, Helicone, Braintrust and Arize Phoenix store traces, show them on a timeline and track cost. Many accept OpenTelemetry, an open standard for traces with rules for AI calls, so you aren't locked in to one tool.
import json import time import requests start = time.perf_counter() response = requests.post("https://api.anthropic.com/v1/messages", headers={"x-api-key": "added-by-the-academy-server", "anthropic-version": "2026-10-21"}, json={"model": "claude-opus-5-5", "max_tokens": 100, "messages": [{"role": "user", "content": "Suggest a subject line for a rain jacket sale."}]}, timeout=60) latency_ms = round((time.perf_counter() - start) * 1000) body = response.json() record = {"feature": "subject_lines", "prompt_version": "v3", "model": body["model"], "id": body["id"], "latency_ms": latency_ms, "stop_reason": body["stop_reason"], **body["usage"]} print(json.dumps(record, indent=1))
How it works:
time.perf_counter()reads a precise clock before and after the call. The difference, times 1,000, is the latency in milliseconds.**body["usage"]copies every usage field into the record, so the log line includes all the token counts.json.dumps(..., indent=1)turns the record into readable JSON text.
Without a trace, a bad answer is a mystery. With one, it's a lookup.
Show the answer
The reply's id (and the request-id header)
The id points at one specific call. The other fields help you understand it, but only the id identifies it.
Keep personal data out
Prompts carry customer names, emails, order details and sometimes worse. Logs get copied, shared and kept for years. Before you write a log line:
- Redact obvious personal data: hide emails, phone numbers, card numbers and addresses.
- Keep it short-term. Keep full text for days or weeks, and only totals for longer.
- Limit access. Logs of customer conversations deserve the same care as your CRM.
Simple patterns catch the common cases:
import re text = "Hi, I'm Sam (sam.ortiz1987@mail.example, +1 503-555-0142). Order 5521 hasn't arrived." text = re.sub(r"[\w.+-]+@[\w-]+\.[\w.-]+", "[email]", text) text = re.sub(r"\+?\(?\d[\d\s().-]{7,}\d", "[phone]", text) print(text)
It prints Hi, I'm Sam ([email], [phone]). Order 5521 hasn't arrived. The order number stays, because four digits are too short to look like a phone number.
Emails are handled first, so digits inside an email address aren't mistaken for a phone number. Patterns like these sometimes hide too much. A date such as 2026-10-25 looks like a phone number, for example. That's the safe direction to be wrong in. Dedicated tools for finding personal data do better on names and addresses.
The numbers worth watching
Logs become useful when you sum them up every day:
- Cost per day, by feature and by model.
- Latency percentiles. The average hides the slow requests your users complain about. Track the median (p50), the middle value, and the 95th percentile (p95): the time that 95% of requests beat.
- Error rate, split by status code. Rate limits are a capacity problem; 400s are a bug.
- Cut-off rate: the share of replies with
stop_reasonmax_tokens. - Cache hit rate: cache-read tokens as a share of all input.
- Feedback rate: thumbs-down per hundred replies.
Set alerts on the ones that matter, such as an error rate above 2% or a daily cost 50% above normal. And once a week, read a random sample of real conversations. Add the bad ones to your eval set.
For marketers: this is your analytics stack for AI features. p95 latency is like page-load time for your slowest visitors, and the thumbs-down rate is your bounce rate.
times = sorted([120, 130, 140, 150, 2000]) print(times[len(times) // 2])
Show the answer
140
There are 5 times, so len(times) // 2 is 2, and position 2 in the sorted list is 140. The median ignores the one slow call. The average would hide it, which is why percentiles matter.
Why the average hides the slow tail
Ten requests took these many seconds. Nine are quick; one took 9.5 seconds. Work out the average, the median (the middle value once sorted) and the p95 (the value 95% of requests beat).
import math latencies = [1.2, 1.4, 1.5, 1.6, 1.8, 2.0, 2.1, 2.4, 3.0, 9.5] s = sorted(latencies) print(round(sum(s) / len(s), 2)) print(s[len(s) // 2]) print(s[math.ceil(0.95 * len(s)) - 1])
It prints 2.65, 2.0 and 9.5. The average of 2.65 seconds looks fine. The median of 2.0 says a typical request is quick. But the p95 of 9.5 seconds shows that some users wait almost 10 seconds. That's the person who writes the complaint. (math.ceil rounds up: 95% of 10 requests is 9.5, rounded up to the 10th.)
Show the answer
The p95 latency: a slow tail can hide behind a fine average
Percentiles show what the slowest group of users experiences.
Show the answer
About 1 in 12 answers is being cut off partway; raise max_tokens or ask for shorter answers
max_tokens means the reply hit your length limit before it finished. Users are seeing truncated answers.
Quiet logs don't mean good answers
An answer can be wrong, and still return a clean 200 with no error logged. So logs alone can't tell you about quality. That's what evals are for.
Practice what you just read
- 3 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 4 review cards for spaced repetition
Try a quiz question
Why are LLM failures hard to spot without logging?