What are entropy and information gain in a decision tree?
Entropy measures how mixed a group is: 0 when pure, 1 for a 50-50 split of two classes. Information gain is the entropy before a split minus the weighted entropy after. The tree picks the split with the highest gain.
Explain it simply
Entropy is "how surprised would I be by a random pick from this group?" A basket of only mangoes: no surprise, entropy 0. A basket half mangoes, half apples: maximum surprise for two kinds of fruit, entropy 1.
A good question splits the basket into groups that are less mixed than before.
Example
Formula: entropy = - sum of p x log2(p).
Eight students, 4 placed and 4 not placed:
Entropy before = -(0.5 x log2 0.5 + 0.5 x log2 0.5) = 1.000Split on "has an internship":
Yes group: 4 students, 3 placed, 1 not
entropy = -(0.75 x log2 0.75 + 0.25 x log2 0.25)
= 0.75 x 0.415 + 0.25 x 2 = 0.811
No group: 4 students, 1 placed, 3 not
entropy = 0.811 (same by symmetry)
Weighted after = 0.5 x 0.811 + 0.5 x 0.811 = 0.811
Information gain = 1.000 - 0.811 = 0.189The tree compares this gain against every other possible split and takes the largest.
Scikit-learn uses Gini by default (criterion="gini"); criterion="entropy" switches to this. Results are usually very similar.
What to say in the interview
"Entropy measures impurity, and information gain is the drop in entropy after splitting, weighted by group sizes. A greedy tree picks the highest-gain split at each node. Gini is a close alternative and is the scikit-learn default."
Common mistakes
- Forgetting to weight child entropies by group size.
- Thinking entropy and Gini give very different trees in practice.