Interview Q&A / Data Science

1. What is hypothesis testing, and what does a p-value actually mean?

   Hypothesis testing checks whether data is surprising under a default assumption called the null hypothesis. The p-value is the probability of seeing results at least as extreme as yours if the null hypothesis were true. A small p-value is evidence against the null. It is not the probability that the null is true.

   AdvancedIn-depth5 min read
2. How do you decide whether an A/B test result on conversion rate is significant?

   Compare the two conversion rates with a two-proportion z-test: z is the difference in rates divided by the pooled standard error. A p-value below your preset alpha, with a practically useful lift, supports shipping the change.

   IntermediateQuick answer1 min read
3. Why is peeking at an A/B test or running many tests a problem?

   Each extra look or extra comparison gives another chance of a false positive. With alpha 0.05, 20 independent tests give about a 64% chance of at least one false hit. Fix sample size in advance and correct for multiple comparisons.

   AdvancedQuick answer1 min read