Glossary · Phase 2 · Math, when you need it
Statistical significance
Evidence that a difference you measured (say, in an A/B test) is probably real, not luck.
Try it now
Run the test once, then again a few more times. With a real 20% lift and 5,000 visitors each, some runs come out significant and some don't. 'Run it 500 times' shows the test's power: the share of runs that catch the lift. Then set the lift to 0 and see how often noise looks like a win.
Learn it properly
A/B tests and significance · Tell a real lift from noise. About 30 minutes.
Other terms from the same lesson
- Bonferroni correction: When you run many tests, divide the 0.05 cutoff by how many you ran, so chance alone doesn't hand you a fake winner.
- null hypothesis: The boring explanation you try to rule out: "there's no real difference, it's just chance".
- statistical power: The chance a test spots a real difference when one exists. More data gives more power.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.