AI engineer interview questions and answers
Questions about building with language models in production: choosing a model, RAG, evals, cost, safety, and designing a feature end to end.
1. Design an AI feature that drafts replies to customer emails for support agents. Cover latency, cost, failures and guardrails.
Drafts are made in the background when an email arrives, via a queue, so they're ready when an agent opens it. Prompt caching covers the long policy prefix; retrieval adds only the relevant help articles; simple emails go to a small model. Errors retry, then fall back to a second model behind a circuit breaker; if all fail, the agent writes from scratch. The email is treated as untrusted data, never instructions. Drafts are checked for promises like refunds, and a person always reviews and sends. An eval set of past emails gates changes, and we track how much agents edit drafts.
2. In your own words, explain why a model can state something false with complete confidence.
A model writes by repeatedly picking a likely next token. Training makes it good at producing text that sounds right, and a specific, confident claim often sounds right whether or not it's true. Nothing checks the claim against a source unless my app gives it one, so the confident tone tells me nothing about accuracy. I should give it the facts and verify what matters.
3. A stakeholder says: 'Let's fine-tune a model on our 300-page product manual so it knows our products.' What would you recommend instead, and why?
I'd build retrieval over the manual first: split it into sections, find the relevant ones for each question, and have the model answer from them with references. That gives accurate answers that update the moment the manual changes, with sources we can check. Fine-tuning on the manual would teach the model to sound like it knows our products, but it learns facts unreliably, makes confident mistakes, and would need retraining after every manual update. If we later want answers in a particular format or tone, a small fine-tune on top can help, but only after an evaluation shows retrieval and prompting aren't enough.
4. A colleague says "RAG means the model learns our documents." Correct them in two or three sentences.
RAG doesn't change the model at all. For each question, our code searches the documents, pastes the most relevant passages into the prompt, and the model answers from what it was given, so if retrieval misses the right passage, the answer will be wrong or missing.
5. Your model accepts up to a million tokens per request. Give two reasons not to send everything you have, every time.
First, cost: I pay for every input token on every call, so unneeded material multiplies the bill across thousands of requests. Second, speed and focus: more input takes longer to process, and an important detail is easier to miss when it's buried in irrelevant text. I'd send the material the task needs and leave room for the reply.
6. Why shouldn't you keep editing a prompt until every example in your eval passes?
Because the prompt ends up tailored to those exact examples, so a perfect score no longer tells you how it'll do on new inputs. Keep some examples aside as a held-out set you only check at the end, like a holdout group in a campaign test.
7. Explain why many production apps use more than one model.
Not every step is equally hard. Routing easy, high-volume work like tagging to a small fast model and saving the strong model for the hard steps keeps quality where it matters while cutting cost and latency overall.
8. A colleague proposes five cooperating agents to write the weekly newsletter (planner, researcher, writer, editor, publisher). What would you ask or suggest?
The newsletter has a known sequence, so I'd start with a workflow: gather the inputs, draft, then an evaluator-optimizer loop against our voice guide. Five agents add handoffs, cost and unpredictability without independent work to split up. I'd measure the simple version against a few past issues first, and keep publishing behind a person's approval.
9. Explain the lethal trifecta and one design change that breaks it for an AI assistant that reads customer emails and your CRM.
An agent is dangerous when it can read private data, sees untrusted content and can send data out, because instructions hidden in the untrusted content can make it ship the private data somewhere. For an email assistant with CRM access, I'd remove the outbound leg: it writes drafts and a person sends them, and it can't make web requests.
10. Structured output guarantees the reply matches your schema. Why should your code still validate the data before acting on it?
The schema only promises the right fields and types. It can't say a discount must be 5 to 50 percent, that a not-a-fit lead shouldn't get a demo, or that a company name isn't blank, and the model can fill valid fields with wrong values. Replies can also be cut off or refused. So I check stop_reason, parse, then run my own rule checks before anything acts on the data.
11. Explain why the model never executes your tools itself, and why that matters for safety.
With your own tools the model only produces a request: which tool and what inputs. Your code chooses whether to run it. That means you decide which functions are exposed, you can validate every input, and you can require a person's approval before anything that sends, spends or deletes.
12. Before using an LLM judge to approve 500 product descriptions, how would you check it's trustworthy?
I'd label 40 descriptions myself against the rubric, run the judge on the same 40 and measure how often we agree. If agreement is low I'd read the disagreements, make the vague criteria more specific or give the judge the guide it needs, and measure again. Once it's around 0.9, I'd still spot-check a random sample of its passes.
13. Your product-description generator costs too much. List the levers you'd try, in order, and how you'd make sure quality doesn't drop.
First cache the shared instructions and brand guide, which are identical for every product. Then test a cheaper model and lower effort, cap the output length, and move the nightly catalog run to the Batch API for half price. I'd run our eval of 50 labeled descriptions before and after each change and keep only the ones that don't lower the score.
14. Your support bot launched last week. Describe what you'd monitor and how you'd use it to improve the bot.
I'd track daily cost, p50 and p95 latency, error rate, the share of truncated replies and thumbs-down rate, with alerts if errors or cost spike. Every conversation would be logged with personal data redacted, so I can open a bad one and see what was retrieved and said. Each week I'd read flagged and random conversations and add the failures to our eval set before changing the prompt.
15. List the protections you'd put around a public "ask our AI" box on your website before launch, and why each matters.
Keep the key on the server so nobody can steal it; rate-limit per visitor and cap input length so one person or bot can't run up costs; set max_tokens and a monthly spend limit with alerts; show a friendly message and fall back to the help center when the model is busy; log conversations with personal data redacted; and test grounding and injection examples in our eval before launch.
16. A stream has delivered half a reply when an error event arrives. Why can't your code rely on the HTTP status code here, and what should it do?
With streaming, the server sends 200 OK at the start, before the model has written anything, so a mid-stream failure can't show up in the status. It arrives as an error event. My loop has to check for that event, treat the text so far as incomplete rather than using it, and then retry or tell the user something went wrong.
17. You need to classify 20,000 old reviews by next week. Describe how you'd run it and why.
It isn't urgent, so I'd use the Message Batches API: half the price, results within a day, each request tagged with the review id as custom_id. I'd test the prompt on 50 labeled reviews first, use structured output so results are machine-readable, and keep track of finished ids so I can resubmit only the failures.
18. Your legal team says customer survey responses must never be sent to a third-party AI service. You need them tagged by theme. Propose a setup and its trade-offs.
I'd run an open-weights model on our own hardware, for example a 7–8B model at 4-bit through Ollama on a workstation, or a larger one on a company GPU server, after confirming its license allows commercial use. The survey text never leaves our machines. Before trusting it, we'd hand-label a few hundred responses and check how well the model's tags match, improving the prompt with examples or fine-tuning if needed. The trade-offs: it will be less capable than a frontier hosted model and we take on setup and maintenance, but we get privacy, no per-token cost, and full control over the model version.
19. Your team is building an AI agent that can read the shared inbox and reply to customers. List the safeguards you'd put in place and why.
I'd give it the narrowest access that does the job: read the inbox and draft replies, but no ability to issue refunds, delete messages or send in bulk. At first, every reply would need a person to approve it, and later only routine categories would go out automatically. Because anyone can email us, the agent must treat email text as information, never as instructions, so I'd test it with emails that try to hijack it. Finally I'd log every action, review samples weekly, track mistakes like wrong policy promises, and keep a switch to turn it off.
20. A vendor shares a paper claiming their fine-tuning method 'beats the baseline on all five benchmarks'. What would you check in a 20-minute second pass?
First, the baseline: is it a strong, current method run with the same care and compute, or a weak one? Second, whether the gains exceed the noise: how many test items per benchmark, how many runs, any confidence intervals, and whether the benchmarks could have leaked into training. Third, the ablations and scope: which part of the method causes the improvement, what it costs in compute, what the limitations section admits, and whether any of the five benchmarks resemble our actual tasks. 'Beats on all five' can mean five small, noisy wins over a weak baseline.
21. Your app summarizes uploaded PDFs with a model, which takes 20 seconds. Users currently wait on the request. Redesign it with a queue and a cache.
The upload endpoint stores the file, pushes a job and returns 202 with a job id at once. Workers pull jobs with a visibility timeout longer than a model call, say 2 minutes. They call the model with retries and backoff, and send repeated failures to a dead-letter queue. Results are saved under the job id; the page polls a status endpoint or gets a notification. Workers are idempotent, keyed by job id. Summaries are cached by the file's hash plus the model and prompt version, so a repeat upload is instant.
22. You review an AI-generated pull request that adds a caching layer. Its tests pass. What would you still check, and why?
Passing tests only prove what they test. I'd check the cache keys include everything that changes the answer, like the user id, so nobody sees another user's data. I'd check how entries are invalidated when data changes, and what happens on a cache outage. I'd read the tests to see they'd fail if caching were wrong, not only that the code runs. And I'd check any new dependency is a real, maintained package, and that the design fits the rest of the codebase.