Glossary · Phase 11 · Professional practice
Novelty effect
Extra attention anything new gets in its first days, which fades and can make a change look better early on than it really is.
Two more traps are about time. A novelty effect makes anything new get extra clicks for its first days, and it fades. And weekdays differ from weekends, so run whole weeks: a test that runs Monday to Friday measures weekday users only.
Try it now
A guardrail stops a launch; a validity check stops you reading the results at all.
Learn it properly
A/B testing a model change · Hash assignment, per-user results from raw logs, sample ratio checks, guardrails and a decision rule. About 40 minutes.
Other terms from the same lesson
- guardrail metric: A number an experiment must not make worse, such as response time or cost; breaking its limit blocks a launch whatever the main result.
- peeking: Checking an experiment's result repeatedly and stopping at the first significant one, which makes false wins far more likely than the stated 5%.
- primary metric: The one number an experiment is designed to move, chosen before it starts, which the launch decision hangs on.
- sample ratio mismatch: Group sizes in an experiment that are far from the planned split, a sign that something treated the groups differently, so no result can be trusted until it's explained.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.