How do you detect outliers using the IQR and z-score methods?
With the IQR method, a value is an outlier if it lies below Q1 minus 1.5 times the IQR or above Q3 plus 1.5 times the IQR. With the z-score method, a value is flagged if it is more than about 3 standard deviations from the mean. IQR is more robust for skewed data and small samples, because outliers inflate the mean and standard deviation.
Explain it simply
An outlier is a value that does not fit with the rest. If a class test is out of 100 and most students score between 12 and 20 out of 25, a score of 95 is either a data entry error (someone typed it out of 100) or something genuinely unusual. Both are worth noticing.
The two common methods draw a "normal range" fence around the data and flag anything outside it. They just build the fence differently.
Example
Marks of 10 students: 12, 14, 15, 15, 16, 17, 18, 19, 20, 95
IQR method (using pandas' default quartile calculation):
- Q1 (25th percentile) = 15, Q3 (75th percentile) = 18.75
- IQR = Q3 - Q1 = 3.75
- Lower fence = 15 - 1.5 × 3.75 = 9.375
- Upper fence = 18.75 + 1.5 × 3.75 = 24.375
- 95 is above 24.375, so it is an outlier.
Z-score method:
- Mean = 241 / 10 = 24.1
- Standard deviation (dividing by n) ≈ 23.74
- z for 95 = (95 - 24.1) / 23.74 ≈ 2.99
With the usual cut-off of 3, the z-score method does not flag 95. The outlier dragged the mean from about 16 up to 24.1 and blew up the standard deviation, so it partly hid itself. This is called masking, and it is why the IQR method is often preferred.
import pandas as pd
marks = pd.Series([12, 14, 15, 15, 16, 17, 18, 19, 20, 95])
q1, q3 = marks.quantile(0.25), marks.quantile(0.75)
iqr = q3 - q1
iqr_outliers = marks[(marks < q1 - 1.5 * iqr) | (marks > q3 + 1.5 * iqr)]
print(iqr_outliers.tolist()) # [95]
z = (marks - marks.mean()) / marks.std(ddof=0)
print(marks[z.abs() > 3].tolist()) # [] (z for 95 is about 2.99)How it works
- IQR (interquartile range) is the spread of the middle 50% of the data. Quartiles depend only on order, so a few extreme values do not move them much. The 1.5 multiplier is a convention from John Tukey's box plot; the points beyond the whiskers of a box plot are exactly these outliers.
- Z-score says how many standard deviations a value is from the mean. It works well when the data is roughly normal and the sample is large. Because it uses the mean and standard deviation, which outliers affect, it is less reliable for small or skewed data.
- Visual checks matter too: box plots, histograms and scatter plots often show problems faster than a formula.
Real-world use
- Fraud detection: a ₹2 lakh transaction on a card that usually spends ₹2,000 a week.
- Data cleaning: an age of 250 or a height of 17 cm is almost certainly an entry error.
- Sensors: a temperature spike from a faulty reading.
- Before training a model: linear regression and k-means are pulled strongly by outliers.
Common mistakes
- Deleting every outlier automatically. Some are the most important rows in the data, such as fraud cases or a genuinely viral post.
- Using z-scores on heavily skewed data such as incomes, where many valid high values look extreme.
- Forgetting that outliers distort the very mean and standard deviation used to find them.
- Not checking with the domain: a ₹5 crore salary is an outlier in a fresher dataset but normal for a CEO dataset.
What to say in the interview
"I first look at the data visually with a box plot or histogram. For a rule-based check, I use the IQR method: anything below Q1 minus 1.5 IQR or above Q3 plus 1.5 IQR is flagged. It is robust because quartiles are not pulled by extreme values. For roughly normal data with plenty of rows, a z-score above 3 also works. Then I decide what to do case by case: fix it if it is an entry error, keep it if it is genuine, or cap it or use a robust model if it would distort training."
Follow-up questions
- Why is the IQR method more robust than the z-score method?
- What would you do with an outlier once you have found it?
- What is winsorising (capping)?
- How would you detect outliers when you have many columns at once?
Go deeper
Read these once the basics above feel easy.
The modified z-score replaces the mean with the median and the standard deviation with the median absolute deviation (MAD). It is much more resistant to masking. A common rule flags values whose modified z-score is above 3.5.
Options after detection: correct the value if you can trace the error, remove it, cap it at a percentile such as the 1st and 99th (winsorising), apply a log transform to reduce skew, or use models that are less sensitive to outliers, like tree-based models or regression with a robust loss.
There is a hard limit on how large a z-score can be in a small sample. Using the population standard deviation, no value in a set of n numbers can have |z| greater than √(n - 1). For n = 10 that is exactly 3, so a "z > 3" rule can never fire on 10 points, no matter how extreme the value. This is why z-score rules need a reasonable sample size.
For many columns together, univariate fences miss points that are normal in each column but unusual in combination (a 20-year-old with 30 years of work experience). Methods such as Mahalanobis distance, Isolation Forest and Local Outlier Factor handle this multivariate case.