How do you choose k in KNN, and what happens when k is very small or very large?
Small k (like 1) follows noise and overfits; very large k smooths too much and underfits, at the extreme predicting the majority class for everything. Choose k by cross-validation, and use an odd k for two-class problems to avoid ties.
Explain it simply
Asking just your single nearest neighbour is risky: one grumpy person can mislead you. Asking the entire city is useless: you just get the city's average opinion. The right number is somewhere in between.
Example
k = 1 : boundary wiggles around every point, training accuracy 100%
(each point is its own nearest neighbour) -> high variance
k = N : every query gets the same answer, the majority class
-> high bias
k in between: smoother boundary, best test scoreIf the data has 100 rows, 70 of class A and 30 of class B, then k = 100 predicts A for every query: 70% accuracy and no useful model.
from sklearn.model_selection import GridSearchCV
from sklearn.neighbors import KNeighborsClassifier
grid = GridSearchCV(KNeighborsClassifier(),
{"n_neighbors": [1, 3, 5, 7, 9, 11, 15]}, cv=5)
grid.fit(X_train, y_train)
print(grid.best_params_)A common starting point is k near the square root of the number of rows, but treat it only as a first guess and let cross-validation decide.
What to say in the interview
"Small k gives low bias and high variance, large k the reverse. I pick k with cross-validated grid search on scaled features, prefer odd values in binary problems, and may weight neighbours by distance."
Common mistakes
- Judging k by training accuracy, where k = 1 always looks perfect.
- Trusting the square-root-of-n rule blindly.