Rejection-sampling fine-tuning
Generating many answers, keeping only the ones that pass a check, and fine-tuning on those.
What it means
Rejection-sampling fine-tuning is a cheap way to get training data for a model you want to improve. You have the model generate many answers, reject the ones that fail a check, and fine-tune on what's left. You don't need people to write examples, only a way to check them.
The selection step in Python
answers = [
"The Lark tent is roomy. Backed by our repair promise.",
"The Lark tent is roomy and roomy. Backed by our repair promise.",
"The Lark tent is made from recycled nylon.",
"The Heron mug is roomy. Backed by our repair promise.",
"The Lark tent packs small. Backed by our repair promise.",
]
def passes(answer):
words = answer.lower().replace(".", "").split()
no_repeats = all(a != b for a, b in zip(words, words[2:]))
return "Lark" in answer and "repair promise" in answer and no_repeats
keep = [a for a in answers if passes(a)]
print(f"kept {len(keep)} of {len(answers)} samples to fine-tune on:")
for answer in keep:
print(" ", answer)
kept 2 of 5 samples to fine-tune on: The Lark tent is roomy. Backed by our repair promise. The Lark tent packs small. Backed by our repair promise.
Three samples fail: one repeats a word, one lacks the brand's promise, and one describes the wrong product. The two that pass become training data.
The catch
The model learns whatever your checker rewards, so a weak checker teaches the wrong thing. It also trains the model on its own outputs, which strengthens its habits, good and bad. In the course's project, a model tuned this way cut brand violations by 29% and got worse at naming the right product, which is why you test with release gates.
Don't confuse it with rejection sampling in statistics, a method for drawing samples from a hard distribution. Both throw samples away, but they have different goals.
Try it now
The same checklist works on a tiny GPT and on a real open model.
Learn it properly
Fine-tune an open model · Run a real fine-tune and evaluate the result. About 45 minutes.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.