How do you stop a decision tree from overfitting?
Limit its growth (max_depth, min_samples_leaf, min_samples_split, max_leaf_nodes) or prune it afterwards with cost-complexity pruning (ccp_alpha). Choose the settings by cross-validation. Or use an ensemble such as a random forest.
Explain it simply
An unrestricted tree keeps asking questions until each leaf holds one student. It then "knows" that the student with roll number 47 who wears a blue shirt got placed. That memorises the past instead of learning a rule.
A depth of d allows up to 2^d leaves. Depth 3 gives at most 8 leaves. Depth 10 gives up to 1,024, enough to isolate individual rows in a small dataset.
Example
from sklearn.tree import DecisionTreeClassifier
tree = DecisionTreeClassifier(
max_depth=4, # at most 2^4 = 16 leaves
min_samples_leaf=20, # each leaf must have 20+ rows
random_state=0,
)Pre-pruning (stop early): max_depth, min_samples_leaf, min_samples_split, max_leaf_nodes.
Post-pruning: grow a full tree, then remove branches that add little. ccp_alpha sets the penalty per leaf; larger values give smaller trees.
How to tell it is overfitting:
Train accuracy 100%, validation accuracy much lower -> overfit
Both low -> too shallowPick settings with GridSearchCV, comparing validation scores, not training scores. If a single tree is still unstable, bagging many trees (random forest) lowers variance.
What to say in the interview
"A fully grown tree overfits, so I constrain depth and leaf size or use cost-complexity pruning and choose values by cross-validation. If a single tree remains unstable, I move to a random forest or gradient boosting."
Common mistakes
- Tuning on training accuracy.
- Setting max_depth very high and hoping the rest will cope.