What are Type I and Type II errors in hypothesis testing?
A Type I error is a false positive: rejecting a true null hypothesis, with probability alpha. A Type II error is a false negative: failing to reject a false null, with probability beta. Power is 1 - beta.
Explain it simply
Think of a smoke alarm. A Type I error is the alarm ringing when there is no fire. A Type II error is silence when there is a fire.
In a hypothesis test the null hypothesis (H0) says "nothing is going on", such as the new design has no effect.
| H0 actually true | H0 actually false | |
|---|---|---|
| Reject H0 | Type I error (false alarm) | correct, called power |
| Do not reject H0 | correct | Type II error (missed effect) |
You choose alpha, usually 0.05, so a Type I error happens 5% of the time when H0 is true. Beta depends on sample size, effect size and noise.
Example
A drug trial tests whether a new medicine works.
- Type I: the trial says it works, but it does not. Patients take a useless medicine.
- Type II: the trial says no effect, but it works. A good medicine is dropped.
Lowering alpha to 0.01 reduces false alarms but raises beta unless you increase the sample. The two errors trade off.
alpha, beta = 0.05, 0.20
print(round(1 - beta, 2)) # 0.8 (power)What to say in the interview
"Type I is rejecting a true null, controlled by alpha. Type II is missing a real effect, with probability beta, and power is 1 minus beta. To reduce both at once, I need more data. Which error is worse depends on the cost: for fraud screening a missed case may cost more than a false flag."
Common mistakes
- Thinking alpha is the probability that the null is true.