What is the difference between L1 and L2 regularisation?
Both add a penalty on weight size to the loss to reduce overfitting. L1 (Lasso) adds the sum of absolute weights and pushes many weights to exactly zero, so it also selects features. L2 (Ridge) adds the sum of squared weights and shrinks all weights smoothly towards zero without removing them.
Explain it simply
Regularisation is a tax on complexity. The model still tries to fit the data, but now every large weight costs extra. So the model only keeps a big weight if it truly helps predictions.
The two taxes behave differently:
- L1 charges the same rate for every rupee of weight, big or small. Small, barely useful weights are not worth paying for, so the model sets them to exactly zero. You end up with fewer features.
- L2 charges in proportion to the square. Big weights are taxed heavily, small weights hardly at all. So the model shrinks big weights a lot but rarely bothers to make any weight exactly zero.
Example
Linear regression loss with each penalty, where λ (lambda) controls the strength:
- L1 (Lasso): Loss = MSE + λ × (|w₁| + |w₂| + … + |wₙ|)
- L2 (Ridge): Loss = MSE + λ × (w₁² + w₂² + … + wₙ²)
Suppose you predict crop yield from 50 features: rainfall, fertiliser, soil pH, and many weak or duplicate ones like rainfall in mm and rainfall in cm.
| No regularisation | L2 (Ridge) | L1 (Lasso) | |
|---|---|---|---|
| Weights | Large, unstable | All smaller, none zero | Many exactly zero |
| Duplicate rainfall columns | Large opposite weights that cancel | Weight shared roughly equally | One kept, the other often zero |
| Features used | 50 | 50 | Perhaps 8 |
from sklearn.linear_model import Ridge, Lasso, ElasticNet
ridge = Ridge(alpha=1.0) # L2
lasso = Lasso(alpha=0.1) # L1
enet = ElasticNet(alpha=0.1, l1_ratio=0.5) # mix of bothIn scikit-learn, alpha plays the role of λ. Scale the features first (for example with StandardScaler), because the penalty treats all weights equally, and a feature measured in lakhs would otherwise need a tiny weight and escape the penalty.
How it works
Why L1 gives exact zeros. Look at the gradient of the penalty. For L2 it is 2λw, which gets smaller as w gets smaller, so the push towards zero fades away and w settles at a small non-zero value. For L1 it is λ × sign(w), a constant push of the same size no matter how small w is. If a feature's contribution to reducing the error is weaker than that constant push, the weight is driven all the way to zero and stays there.
The geometric picture. Minimising loss with a penalty is like minimising loss inside a budget region for the weights. For two weights, the L2 region is a circle and the L1 region is a diamond with corners on the axes. The loss contours usually first touch the diamond at a corner, where one weight is zero. A circle has no corners, so the touching point usually has both weights non-zero.
Effect on bias and variance. Both penalties add a little bias (weights are pulled away from their unregularised values) in exchange for lower variance. Choose λ with cross-validation: too small does nothing, too large underfits.
Real-world use
- L1 when you suspect only a few features matter and want an interpretable model, for example picking a handful of genes or survey questions out of hundreds.
- L2 when many features each contribute a little, or when features are correlated. It is the default in many tools, including scikit-learn's
LogisticRegression. - Elastic Net combines both: it can zero out features like L1 while handling groups of correlated features more stably, like L2.
- In neural networks, L2 appears as weight decay. With Adam, the decoupled version (AdamW) is preferred.
Common mistakes
- Forgetting to scale features before regularising.
- Penalising the bias (intercept) term. Standard implementations do not, and they should not.
- Trusting the exact features Lasso keeps when features are highly correlated. It may pick one arbitrarily, and the choice can change with a small change in data.
- Mixing up the direction of
Cin logistic regression.Cis the inverse of regularisation strength, so a smallerCmeans a stronger penalty.
What to say in the interview
"Both add a penalty on the size of the weights to the loss, trading a little bias for lower variance. L1, or Lasso, adds lambda times the sum of absolute weights. Its gradient is a constant push towards zero, so many weights become exactly zero, which gives sparse models and built-in feature selection. L2, or Ridge, adds lambda times the sum of squared weights. It shrinks all weights smoothly but rarely to zero, and it handles correlated features better by sharing weight between them. Elastic Net mixes both. I scale features first and tune lambda with cross-validation."
Follow-up questions
- Why does L1 regularisation produce sparse weights?
- What happens as lambda goes to zero or to infinity?
- How is L2 regularisation related to weight decay in neural networks?
- When would you choose Elastic Net over Lasso?
Go deeper
Read these once the basics above feel easy.
From a Bayesian view, L2 regularisation is the same as putting a Gaussian prior on the weights and finding the most probable weights (MAP estimate), while L1 corresponds to a Laplace prior, which is sharply peaked at zero. Ridge regression also has a closed-form solution, w = (XᵀX + λI)⁻¹Xᵀy, and adding λI makes the matrix invertible even when features are perfectly correlated. Lasso has no closed form because |w| is not differentiable at zero, so it is solved with coordinate descent or subgradient methods. When the number of features is larger than the number of rows, Lasso can select at most as many features as there are rows, which is one reason Elastic Net was proposed.