What is hypothesis testing, and what does a p-value actually mean?
Hypothesis testing checks whether data is surprising under a default assumption called the null hypothesis. The p-value is the probability of seeing results at least as extreme as yours if the null hypothesis were true. A small p-value is evidence against the null. It is not the probability that the null is true.
Explain it simply
A friend tosses a coin 100 times and gets 60 heads. Is the coin biased?
You start by assuming the boring explanation: the coin is fair. That is the null hypothesis. Then you ask: "If the coin really were fair, how often would I see something this lopsided just by luck?" That probability is the p-value. If it is very small, the boring explanation looks hard to believe, and you reject it. If it is not that small, you do not have enough evidence to call the coin biased.
It works like a court case. The accused is presumed innocent (the null). The prosecution must bring strong evidence. "Not guilty" does not mean "proven innocent"; it means the evidence was not strong enough.
Example
Question: 60 heads in 100 tosses. Is the coin unfair, at the 5% significance level?
- Null hypothesis (H0): the probability of heads is 0.5.
- Alternative (H1): the probability of heads is not 0.5 (two-sided, because a coin biased towards tails would also count).
- Significance level (alpha): 0.05, chosen before looking at the data.
- Compute the p-value: under H0, the chance of 60 or more heads is about 0.028. Doubling for the two-sided test (40 or fewer heads is just as extreme) gives p ≈ 0.057.
- Decide: 0.057 is greater than 0.05, so we fail to reject H0. The result is suggestive, but not strong enough at this level.
from scipy import stats
result = stats.binomtest(60, n=100, p=0.5) # two-sided by default
print(round(result.pvalue, 3)) # 0.057Note how close this is to the line. Sixty heads feels convincing, but fair coins produce results this lopsided roughly 1 time in 18.
How it works
- Null hypothesis: the "no effect" or "no difference" claim. Alternative: what you suspect instead.
- Test statistic: a number summarising how far the data is from what H0 predicts (a count, a t-statistic, a chi-square value).
- p-value: P(data at least this extreme | H0 is true).
- Decision: if p < alpha, reject H0. Otherwise, fail to reject.
- Type I error: rejecting a true H0 (a false alarm). Its probability is alpha.
- Type II error: failing to reject a false H0 (a missed effect). The probability of avoiding it is called power, and it grows with sample size and effect size.
Common tests: a t-test to compare two means, a chi-square test for counts in categories, and a z-test for proportions for conversion rates in A/B tests.
Real-world use
- A/B testing: did the new checkout page really raise the conversion rate, or is the difference noise?
- Medicine: does a drug lower blood pressure more than a placebo?
- Manufacturing: has the average weight of biscuit packets shifted from the labelled 200 g?
Common mistakes
- Saying "p = 0.03 means there is a 3% chance the null is true". The p-value assumes the null is true; it cannot also be the probability of it.
- Saying "p > 0.05 proves there is no effect". It only means the data did not give enough evidence. The sample may be too small.
- Treating a tiny p-value as a large effect. With millions of users, a 0.01% difference can be "significant" and still useless for the business.
- Peeking at an A/B test every day and stopping as soon as p dips below 0.05. This inflates false positives.
- Running many tests and reporting only the significant ones (p-hacking).
What to say in the interview
"Hypothesis testing starts with a null hypothesis, usually 'no effect', and asks whether the data is surprising if that were true. The p-value is the probability of getting results at least as extreme as what we observed, assuming the null is true. If it is below a pre-chosen significance level like 0.05, we reject the null. A p-value is not the probability that the null is true, and a non-significant result does not prove there is no effect. I would also report the effect size and a confidence interval, because statistical significance is not the same as practical importance."
Follow-up questions
- What are Type I and Type II errors? Which one does alpha control?
- What is statistical power and how do you increase it?
- How is a confidence interval related to a hypothesis test?
- Why is stopping an A/B test early a problem?
- What is the multiple comparisons problem and how does the Bonferroni correction address it?
Go deeper
Read these once the basics above feel easy.
A 95% confidence interval and a two-sided test at alpha = 0.05 agree with each other: if the interval for a difference does not include 0, the test rejects "no difference". Intervals are usually more useful to report, because they show both the size of the effect and the uncertainty.
Before an A/B test, decide the minimum effect worth detecting, the alpha and the desired power (often 80%), and calculate the sample size from those. Then run the test to that size without peeking, or use a sequential testing method designed for repeated looks.
When you run m independent tests at alpha = 0.05, the chance of at least one false positive is 1 - 0.95^m, which is already about 40% for m = 10. The Bonferroni correction tests each at alpha / m; it is simple but conservative. The Benjamini-Hochberg procedure controls the false discovery rate instead and keeps more power when many tests are run, which is common in genomics and large-scale experimentation.
A p-value is also only as good as the test's assumptions. A t-test assumes independent observations and roughly normal sampling distributions; clustered data (several orders from the same user) breaks independence and makes p-values look smaller than they should.