What are variance and standard deviation, and why do we need them?
Variance is the average of the squared distances from the mean, and standard deviation is its square root. They measure how spread out the data is. Two datasets can have the same mean but very different spread, and standard deviation is easier to read because it is in the same units as the data.
Explain it simply
The mean tells you the centre of the data. It says nothing about how much the values jump around that centre.
Think of two batsmen who both average 50. One scores close to 50 in every match. The other scores a duck one day and a century the next. Same average, very different players. Standard deviation is the number that tells them apart: small for the steady batsman, large for the unpredictable one.
Example
Last five scores of two batsmen:
- Batsman A:
48, 50, 52, 50, 50 - Batsman B:
0, 100, 10, 90, 50
Both have a mean of 50.
Batsman A: distances from 50 are -2, 0, 2, 0, 0. Squares: 4, 0, 4, 0, 0. Sum = 8. Variance = 8 / 5 = 1.6. Standard deviation = √1.6 ≈ 1.26 runs.
Batsman B: distances are -50, 50, -40, 40, 0. Squares: 2500, 2500, 1600, 1600, 0. Sum = 8200. Variance = 8200 / 5 = 1640. Standard deviation = √1640 ≈ 40.5 runs.
So A typically lands within about 1 run of his average, and B within about 40 runs of his.
import numpy as np
import pandas as pd
b = [0, 100, 10, 90, 50]
print(np.std(b)) # 40.496... (numpy divides by n by default)
print(pd.Series(b).std()) # 45.276... (pandas divides by n - 1 by default)How it works
- Find the mean.
- Subtract the mean from each value to get the deviation.
- Square each deviation, so negatives do not cancel positives and large gaps count more.
- Average the squares. That is the variance.
- Take the square root. That is the standard deviation, back in the original units (runs, rupees, marks).
Population vs sample. If you have every value (the whole population), divide by n. If you have a sample and want to estimate the spread of the larger population, divide by n - 1. This is called Bessel's correction; dividing by n on a sample tends to underestimate the true spread. In the example above, dividing by 4 instead of 5 gives a variance of 2050 and a standard deviation of about 45.3.
Real-world use
- Finance: the standard deviation of returns is a basic measure of how risky a stock or mutual fund is.
- Manufacturing: a factory filling 1-litre milk packets wants a tiny standard deviation in fill volume.
- Exams: knowing the class average is 60 with a standard deviation of 10 tells you a score of 80 is well above most of the class.
- Machine learning: standardisation divides each feature by its standard deviation so features are on a similar scale.
Common mistakes
- Comparing variances across different units. Variance is in squared units (runs², rupees²), which is why people report standard deviation.
- Not knowing that numpy and pandas use different defaults (n vs n - 1) and getting "different" answers.
- Thinking a high standard deviation means bad data. It only means the values are spread out.
- Forgetting that standard deviation is sensitive to outliers, just like the mean.
What to say in the interview
"Variance measures spread: it is the average squared distance of each value from the mean. Standard deviation is the square root of variance, so it is in the same units as the data and easier to interpret. For a sample I divide by n minus 1 instead of n, to get an unbiased estimate of the population variance. For example, two batsmen can both average 50, but the one with a small standard deviation is far more consistent."
Follow-up questions
- Why do we square the deviations instead of taking absolute values?
- Why divide by n - 1 for a sample?
- What does the 68-95-99.7 rule say about a normal distribution?
Go deeper
Read these once the basics above feel easy.
Squaring is chosen over absolute values mainly because it is smooth and works neatly with algebra and calculus, which is why least squares regression is built on it. The mean absolute deviation is a valid alternative that is less affected by outliers.
For data that is roughly normal, about 68% of values fall within one standard deviation of the mean, about 95% within two, and about 99.7% within three. This only holds for bell-shaped data; for skewed data like incomes, do not rely on it.
#statistics#variance#standard-deviation#descriptive-statistics