How does a decision tree work, and why does a random forest usually perform better?
A decision tree splits data with a series of yes/no questions, choosing each split to make the groups as pure as possible. A single deep tree overfits easily. A random forest trains many trees on random samples of rows and features and averages their votes, which reduces variance and usually improves accuracy.
Explain it simply
A decision tree is like a bank officer with a checklist for a loan. "Is monthly income above ₹40,000? If yes, is the credit score above 700? If yes, approve." Each question splits applicants into smaller groups until each group is mostly one answer.
The problem: one officer with a very long, detailed checklist starts making rules for individual past customers. "Approve if income is ₹41,250 and the applicant lives in Pune." That is overfitting.
A random forest is like asking 300 officers, each of whom has seen a different random sample of past applications and is only allowed to look at a few random factors at each step. Each officer makes some mistakes, but their mistakes are different, so the majority vote is far more reliable than any one of them.
Example
Ten past loan applications: 6 approved, 4 rejected. The tree must pick its first question.
Gini impurity measures how mixed a group is: Gini = 1 − Σ pᵢ². For the full group: 1 − (0.6² + 0.4²) = 1 − 0.52 = 0.48.
Suppose the split "income > ₹40,000" gives:
- Left (yes): 5 applications, 5 approved → Gini = 1 − 1² = 0
- Right (no): 5 applications, 1 approved, 4 rejected → Gini = 1 − (0.2² + 0.8²) = 1 − 0.68 = 0.32
Weighted Gini after the split = 0.5 × 0 + 0.5 × 0.32 = 0.16. The impurity dropped from 0.48 to 0.16, a gain of 0.32. The tree tries every feature and threshold and picks the split with the largest drop, then repeats on each side.
How it works
Decision tree
- At each node, test candidate splits and choose the one that most reduces impurity (Gini or entropy for classification, variance or squared error for regression).
- Stop when a node is pure or a limit is reached:
max_depth,min_samples_leaf, and so on. - A leaf predicts the majority class (classification) or the average value (regression).
- Trees need no feature scaling and handle mixed numeric and categorical features well. They are easy to explain, but small changes in data can produce a very different tree. That is high variance.
Random forest
- Bagging (bootstrap aggregating): each tree is trained on a random sample of rows drawn with replacement, the same size as the original data. On average about 63% of unique rows appear in each sample.
- Random feature subsets: at each split, a tree only considers a random subset of features (a common default for classification is the square root of the number of features). This stops every tree from choosing the same strong feature first, so the trees are less correlated.
- Aggregation: majority vote for classification, average for regression.
Averaging many low-bias, high-variance trees whose errors are not strongly correlated lowers variance without raising bias much.
Real-world use
- Credit scoring, fraud detection, customer churn and crop disease prediction from tabular data.
- Random forests are a strong baseline for tabular data: they need little tuning and give feature importance scores.
- Gradient-boosted trees (XGBoost, LightGBM, CatBoost) often beat random forests on tabular data, but need more careful tuning.
Common mistakes
- Growing a single tree with no depth limit and trusting its training accuracy.
- Thinking a random forest cannot overfit at all. Adding more trees does not cause overfitting, but very noisy data and deep trees still can.
- Trusting impurity-based feature importance blindly. It is biased towards features with many unique values; permutation importance is safer.
- Expecting trees to extrapolate. A regression tree cannot predict values outside the range it saw in training.
What to say in the interview
"A decision tree recursively splits the data on the feature and threshold that most reduce impurity, measured by Gini or entropy, until the leaves are pure or a stopping rule is hit. Trees are interpretable and need no scaling, but a deep tree has high variance and overfits. A random forest trains many trees on bootstrap samples of the rows and considers a random subset of features at each split, which makes the trees different from each other. Averaging their predictions reduces variance, so it generalises better than a single tree, at the cost of interpretability and some speed."
Follow-up questions
- What is the difference between Gini impurity and entropy?
- What is the difference between bagging and boosting?
- What is out-of-bag error?
- How would you prune a decision tree?
Go deeper
Read these once the basics above feel easy.
Out-of-bag (OOB) error: each row is left out of roughly 37% of the bootstrap samples. Predicting each row using only the trees that did not see it gives a free validation estimate without a separate hold-out set. Bagging vs boosting: bagging trains trees independently and in parallel to reduce variance; boosting trains trees one after another, each correcting the errors of the previous ones, which mainly reduces bias.