Why do we scale features, and which algorithms need it?
Scaling puts features on comparable ranges. Distance-based and gradient-based models (KNN, SVM, k-means, PCA, neural networks, regularised regression) need it. Tree-based models do not.
Explain it simply
Suppose you compare two students using age (around 20) and annual family income (around 500,000 rupees). A distance formula will be dominated by income: a difference of 10,000 rupees swamps a difference of 5 years. The model effectively ignores age, not because age is useless but because its numbers are small.
Scaling brings each column to a similar range so every feature gets a fair say.
Example
Two students: (age 20, income 400000) and (age 25, income 410000).
Unscaled Euclidean distance
= sqrt((25-20)^2 + (410000-400000)^2)
= sqrt(25 + 100000000)
= about 10000.001 <- age contributes almost nothingAfter standardising, both columns have mean 0 and spread 1, so a gap of one standard deviation counts equally in either.
Need scaling: KNN, k-means, SVM, PCA, logistic and linear regression with regularisation, neural networks (also trains faster with gradient descent).
Do not need it: decision trees, random forests, gradient boosting, because splits depend only on the order of values, not their scale.
What to say in the interview
"Models that use distances or penalise coefficient size are sensitive to feature scale, so I standardise or min-max scale them. Tree-based models are not, since a split on income above a threshold is unchanged by rescaling. I fit the scaler on training data only."
Common mistakes
- Scaling the target column along with features without a reason.
- Fitting the scaler on the full dataset before splitting.