Why is peeking at an A/B test or running many tests a problem?
Each extra look or extra comparison gives another chance of a false positive. With alpha 0.05, 20 independent tests give about a 64% chance of at least one false hit. Fix sample size in advance and correct for multiple comparisons.
Explain it simply
A 5% false alarm rate is per test. Roll a 20-sided die and getting a 1 has a 5% chance on one roll, but if you roll it 20 times, a 1 is likely. Testing 20 metrics, or checking one test every day until it "wins", works the same way.
Peeking is the same problem over time: a p-value is only valid for the pre-planned single look at the planned sample size.
Example
Probability of at least one false positive across k independent tests at alpha = 0.05 is 1 - 0.95^k.
- k = 5: 1 - 0.95^5 = 0.226
- k = 20: 1 - 0.95^20 = 0.642
for k in (5, 20):
print(k, round(1 - 0.95 ** k, 3)) # 5 0.226 / 20 0.642Fixes:
- Bonferroni: use alpha / k per test (0.05 / 20 = 0.0025).
- Benjamini-Hochberg to control the false discovery rate when there are many tests.
- Sequential testing methods designed for continuous monitoring.
- One primary metric chosen before the test starts.
What to say in the interview
"Repeated looks or many metrics inflate the Type I error rate. I pre-register one primary metric, compute the sample size beforehand, and avoid stopping early on a good p-value. If I must test many things, I adjust with Bonferroni or Benjamini-Hochberg, or use a sequential method."
Common mistakes
- Reporting only the metric that turned out significant out of many tried.