What is overfitting and underfitting, and what is the bias-variance trade-off?
Overfitting means the model memorises the training data, including noise, so it does well in training but badly on new data. Underfitting means the model is too simple to learn the pattern at all. The bias-variance trade-off is the balance between these two errors.
Explain it simply
Imagine three students preparing for a board exam.
- One student only reads the chapter titles. They fail both the mock test and the real exam. That is underfitting: too simple to learn anything useful.
- One student memorises last year's paper word for word, including the printing mistakes. They score 100% on that paper but struggle when the real exam asks the same idea in a new way. That is overfitting: they learnt the noise, not the concept.
- One student understands the concepts and practises varied questions. They do well on both. That is a good fit.
A model should learn the general pattern that will hold on data it has never seen, not the quirks of its training set.
Example
Suppose you train a model to predict exam marks from hours studied.
| Model | Training error | Validation error | Diagnosis |
|---|---|---|---|
| Straight line through curved data | High | High | Underfitting (high bias) |
| Smooth curve | Low | Low | Good fit |
| Very wiggly curve that passes through every point | Almost zero | High | Overfitting (high variance) |
The tell-tale sign of overfitting is a big gap: training error is low but validation error is much higher. The sign of underfitting is that both errors are high.
How it works
The error a model makes on new data can be split into three parts:
- Bias: error from wrong assumptions. A straight line cannot follow a curve, however much data you give it. High bias leads to underfitting.
- Variance: error from being too sensitive to the exact training data. Train a deep decision tree on two different samples and you get two very different trees. High variance leads to overfitting.
- Irreducible noise: randomness in the data itself that no model can remove.
Making a model more complex usually lowers bias but raises variance. Making it simpler does the opposite. The trade-off is finding the complexity where total error on unseen data is lowest.
To fix overfitting: get more data, use a simpler model, add regularisation (L1 or L2), use dropout in neural networks, prune decision trees or limit their depth, stop training early, or use ensembles like random forests.
To fix underfitting: use a more flexible model, add better features, reduce regularisation, or train for longer.
Real-world use
- A spam SMS filter that blocks every message containing "offer" has overfit to a few training examples and will block genuine bank messages too.
- Teams plot learning curves (error vs training set size) to decide whether collecting more data will help. More data helps high variance; it does not fix high bias.
Common mistakes
- Judging a model by its training accuracy. Always compare with validation or test performance.
- Assuming more data fixes everything. If the model is underfitting, more data of the same kind will not help.
- Tuning hyperparameters on the test set, which slowly overfits to the test set itself.
What to say in the interview
"Overfitting is when a model learns the training data too closely, including noise, so it has low training error but high error on new data. Underfitting is when the model is too simple, so both errors are high. Bias is error from overly simple assumptions, and variance is error from being too sensitive to the training sample. Increasing complexity lowers bias but raises variance, so we look for the point where validation error is lowest. To reduce overfitting I would use more data, regularisation, a simpler model or early stopping; to fix underfitting I would add features or use a more expressive model."
Follow-up questions
- How do you detect overfitting in practice?
- Why does a random forest overfit less than a single decision tree?
- How does regularisation reduce variance?
- What is early stopping?
Go deeper
Read these once the basics above feel easy.
For squared-error regression, expected test error at a point decomposes exactly as bias² + variance + noise. Modern deep networks complicate the classic U-shaped picture: very large models trained past the point of fitting the training data perfectly can see test error fall again, a pattern called double descent. In interviews, give the classic trade-off first and mention double descent only if asked about deep learning.