How does a decision tree predict a number in regression?
It splits the data to reduce squared error and each leaf predicts the average target of the rows in it. The output is therefore piecewise constant, a staircase, not a smooth line.
Explain it simply
Estimating rent in a locality: group flats by size, then by floor. For each final group, quote the average rent of flats in that group. A new flat falling in the group gets that group's average.
Example
Leaf with five flats, rents in thousands: 30, 35, 40, 45, 50.
Prediction for any new flat in this leaf = (30+35+40+45+50)/5 = 40A split is chosen by how much it lowers the sum of squared errors.
Before split: rents 20, 30, 80, 90
mean = 55, SSE = 35^2 + 25^2 + 25^2 + 35^2 = 1225+625+625+1225 = 3700
Split into {20, 30} and {80, 90}:
left mean 25 -> SSE = 25 + 25 = 50
right mean 85 -> SSE = 25 + 25 = 50
total SSE = 100The split cuts error from 3700 to 100, so it is excellent.
from sklearn.tree import DecisionTreeRegressor
DecisionTreeRegressor(max_depth=3).fit(X_train, y_train)Predictions can only be values built from training averages, so the model cannot predict beyond the range it has seen.
What to say in the interview
"A regression tree picks splits that minimise squared error, and each leaf outputs the mean target of its rows. The result is a step function, easy to interpret but unable to extrapolate and prone to overfitting without depth limits."
Common mistakes
- Expecting a smooth curve from a tree.
- Forgetting that classification trees use impurity while regression trees use variance.