How do you decide whether an A/B test result on conversion rate is significant?
Compare the two conversion rates with a two-proportion z-test: z is the difference in rates divided by the pooled standard error. A p-value below your preset alpha, with a practically useful lift, supports shipping the change.
Explain it simply
A/B testing sends random visitors to version A or B and compares how many convert. Randomising makes the groups alike in everything but the change, so a difference can be credited to the change, or to luck. The test measures whether luck is a reasonable explanation.
Example
Checkout page A: 200 purchases from 5,000 visitors (4.0%). Page B: 250 from 5,000 (5.0%).
- Pooled rate = 450 / 10,000 = 4.5%
- Standard error = sqrt(0.045 x 0.955 x (1/5000 + 1/5000)) = 0.00415
- z = (0.05 - 0.04) / 0.00415 = 2.41
- Two-sided p = 0.016
from math import sqrt
from scipy.stats import norm
p1, p2 = 200 / 5000, 250 / 5000
pp = 450 / 10000
se = sqrt(pp * (1 - pp) * (2 / 5000))
z = (p2 - p1) / se
print(round(z, 2), round(2 * (1 - norm.cdf(z)), 3)) # 2.41 0.016At alpha = 0.05 this is significant. The lift is 1 point, or 25% relative. Whether it is worth shipping depends on engineering cost and the rupee value of that extra 1%.
What to say in the interview
"I fix the metric, alpha and the sample size in advance, randomise users, run the full duration, then use a two-proportion z-test or chi-square. I report the absolute and relative lift with a confidence interval and also check guardrail metrics like refunds or page speed."
Common mistakes
- Stopping the test the moment p dips below 0.05.