LLM interview questions and answers

In each block, attention is the 'communicate' step: every token looks back at earlier tokens and pulls in information from the relevant ones. The MLP is the 'compute' step: each token, on its own, runs what it has gathered through a two-layer network, which holds most of the block's parameters and much of what the model has memorized. Both results are added back onto the token's running vector, and stacking dozens of blocks repeats that gather-then-process cycle.

At temperature 2 the distribution is so flat that unlikely tokens, typos, odd words, off-brand claims, get picked often, and the copy can derail mid-sentence. I'd keep temperature around 0.8–1.0 with top-p around 0.9 to cut the junk tail, and get creativity from the prompt instead: ask for distinct angles, give examples and constraints. Then generate several candidates and pick the best with a quick review or an LLM judge.

Every turn sends the whole conversation back to the model, so each request has more input tokens to prefill: more billed input and a longer wait for the first token. The KV cache for that conversation also keeps growing, using more GPU memory and making each new token attend over more history. To cut cost, cache the stable start of the prompt, such as the system prompt and earlier turns, so the provider reuses its KV cache and bills those tokens at the cached rate; and trim or summarize older turns instead of sending them in full forever.

A million-token window is great for tasks that need whole documents at once, and it can simplify our pipeline when the material fits. But we'd pay for every one of those tokens on every request and wait longer for each answer, unless the same prefix repeats and can be cached. Models also don't use huge contexts perfectly: facts in the middle get missed more often and combining many scattered facts is harder than finding one. And our full knowledge base is larger than a million tokens anyway. I'd test both on our real questions, and likely keep retrieval but give it a bigger budget, retrieving more passages into the long context.

The system prompt is just text placed at the start of the conversation in a marked turn; the model reads it the same way it reads everything else. It carries weight because post-training taught the model to prioritize system instructions, so it usually works, but nothing forces compliance: a long conversation, conflicting instructions or injected text can still override it. So we should test the rules that matter, and enforce the critical ones in our own code instead of trusting the prompt as a guarantee.

Fine-tuning taught the model what our answers look like, the format and tone, but not a reliable memory of every policy detail: it learned from 2,000 examples to sound like a support agent, and where it doesn't know a policy it produces something plausible in the same style. The fix is to give it the real policy at answer time by retrieving the relevant policy text into the prompt, and to train or instruct it to answer from that text. Then we'd test on held-out policy questions and check answers against the documents.

First the model is taught by example, trained on good conversations people wrote. Then people compare pairs of its answers, and a second model, the reward model, learns to predict which one they'd prefer. Finally the chatbot practices: it writes answers, the reward model scores them, and the chatbot is adjusted toward higher scores, with a penalty that stops it from straying too far from how it started. The main risk is that the reward model is only an imitation of human judgment, so pushed too hard the chatbot learns to please the scorer rather than people, for example by becoming wordy or flattering.

We'd take our current support model as the starting point and keep a frozen copy as the reference. For each pair, we compute how likely both models find the better and the worse reply, and train the model to raise the better reply's likelihood and lower the worse one's, relative to the reference, with β limiting how far it moves. It trains like ordinary fine-tuning, with no reward model and no generation during training. The risk is that it learns whatever most easily separates the pairs, such as longer replies or a stock phrase, rather than real helpfulness, and pushed too hard it can get worse at other things. So we'd evaluate on held-out tickets across every criterion and tune β and training length.

With full fine-tuning, each client would get a complete copy of the model, 16 GB apiece for an 8-billion-parameter model, trained on large GPUs and served separately: forty models to host. With LoRA, everyone shares one frozen base model, and each client gets a small adapter trained on their examples, measured in megabytes and quick to train on a single GPU. At serving time we run one base model and attach the right client's adapter to each request. For narrow jobs like matching a voice, adapters usually come close to full fine-tuning, and we'd still check each client's quality on their own examples.

First, fit: the 4-bit weights are about 5 GB, which leaves room on a 16 GB laptop, but long call transcripts need KV cache too, so I'd test the longest calls we expect and measure tokens per second on the actual laptops. Second, quality: I'd run the same 50 real calls through the 4-bit model and through a 16-bit or hosted reference, and compare the summaries, especially names, numbers and action items, since quantization can damage specific skills while overall quality looks fine. Finally I'd confirm the model's license allows our use and that the local setup keeps call data on the laptop, which is a privacy plus.

First I'd confirm the provider's terms allow using its outputs to train our own model for this purpose. Then I'd have the big model write a few thousand descriptions for a representative set of products, have editors approve or fix them, and fine-tune a small model on the approved examples, feeding it the product data each time so facts come from our catalog, not from memory. Before switching, we'd compare the small model's output with the big model's on products held out from training. The risks are that the student copies the teacher's mistakes, can't beat it, may handle unusual products badly, and needs regular re-checking as the catalog changes.

At 9am everyone arrives at once. Requests queue up, and the server packs them into bigger batches, so each user's tokens stream more slowly and the first token takes longer; if the KV cache memory runs out, new requests simply wait. I'd first make sure we use a serving engine with continuous batching and paged KV caches, such as vLLM, and then add a second replica that scales up for the morning peak. Quantizing the model would also free memory for more conversations and speed up each token.

A small, fast model writes a few words ahead, and the big model checks all of them at once, which takes about as long as writing a single word. The big model keeps the words it agrees with, and at the first one it disagrees with, it substitutes its own choice. Because every word that ends up in the answer is one the big model approved or chose itself (and there's a mathematically exact version for when it samples), the answers are the same as without the trick, just two to three times faster on typical text.

A 2-point lead might be noise: on a test of a few hundred questions, each score is uncertain by several points, so I'd want the question count, confidence intervals and whether both models were run the same way by someone independent. Even if the gap is real, benchmark Z may not resemble our work; it could be saturated or partly memorized. So before switching, I'd run both models on our own test set of real tasks, comparing quality, cost and speed, and switch only if X wins clearly there.

I'd ask what the 98% was measured on: examples kept out of training and out of every decision like choosing the checkpoint, and split so they're genuinely new, such as products the model never saw. I'd ask how the score is computed and whether the judge was checked against human ratings. I'd ask for the baseline, because 98% means little if a well-prompted model already gets 95%. And I'd want to read a sample of outputs and see whether anything else, like tone or accuracy, got worse.

The 'experts' are just MLP blocks inside each layer, and a learned router picks a couple of them for every token at every layer based on the token's internal representation. Nobody assigns them topics, and when researchers looked at Mixtral's routing they found it followed things like syntax and token type more than subject areas. You can't call on an expert by asking for it; the prompt only changes which tokens flow through the model. MoE is about getting a big model's capacity at a smaller model's cost per token, so prompt for the marketing task itself.

A small fine-tuned encoder needs a few thousand labeled emails up front, but then classifies millions cheaply and quickly, with consistent results, and can run on modest hardware; changing the topics means relabeling and retraining. Prompting a large model needs no training data and adapts instantly to new topics, but at five million emails a week it's far slower and more expensive. A good path is to use the LLM to bootstrap labels and prototype, then train a small encoder on those labels for production, checking both against a hand-labeled sample.

Scaling laws are an observed pattern, not a law of nature: over many orders of magnitude, more parameters, data and compute have lowered loss along a predictable curve, and labs use that to plan big training runs. But each 10× buys a smaller gain while costing ten times more, high-quality data is limited, and a lower loss doesn't translate neatly into any particular skill. Progress also comes from other sources, like better data, better training after pretraining, and letting models think longer. So steady improvement is a reasonable expectation, but not a guarantee, and not in every capability at the same pace.