What is bootstrap sampling and the out-of-bag score in a random forest?
Each tree trains on a bootstrap sample: n rows drawn with replacement, containing about 63% of the unique rows. The remaining roughly 37% are out-of-bag for that tree and give a free validation estimate.
Explain it simply
Imagine a lottery drum with 100 numbered balls. You draw one, note it, put it back, and repeat 100 times. Some numbers come up twice or thrice, and some never come up. The set you noted is a bootstrap sample.
Example
The chance a given row is never picked in n draws is (1 - 1/n)^n. As n grows this approaches 1/e, which is 0.368.
n = 10 : (0.9)^10 = 0.349
n = 100 : (0.99)^100 = 0.366
n = 1000 : about 0.368
Unique rows in each bootstrap sample ~ 1 - 0.368 = 63.2%Rows left out are out-of-bag (OOB) for that tree. To score a row, use only the trees that never saw it and take their vote. Averaging over all rows gives the OOB score, a cross-validation-like estimate that costs no extra training.
from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(n_estimators=300, oob_score=True, random_state=0)
rf.fit(X_train, y_train)
print(rf.oob_score_)What to say in the interview
"Each tree in a random forest is trained on a bootstrap sample with about 63% unique rows. The other roughly 37% are out-of-bag, so the forest can estimate its generalisation accuracy from them without a separate validation set. I still keep a final test set."
Common mistakes
- Saying the bootstrap sample has 63% of the size; it has the full size but 63% unique rows.
- Using OOB score after tuning on it heavily and treating it as a final test.