Interview Q&A / Data Science

1. What is hypothesis testing, and what does a p-value actually mean?

   Hypothesis testing checks whether data is surprising under a default assumption called the null hypothesis. The p-value is the probability of seeing results at least as extreme as yours if the null hypothesis were true. A small p-value is evidence against the null. It is not the probability that the null is true.

   AdvancedIn-depth5 min read
2. Why is peeking at an A/B test or running many tests a problem?

   Each extra look or extra comparison gives another chance of a false positive. With alpha 0.05, 20 independent tests give about a 64% chance of at least one false hit. Fix sample size in advance and correct for multiple comparisons.

   AdvancedQuick answer1 min read