How does the K-Nearest Neighbours algorithm make a prediction?
KNN stores all training data. To predict, it finds the k training points closest to the new point and takes a majority vote (classification) or an average (regression) of their targets.
Explain it simply
You move to a new colony and wonder whether the local sweet shop is good. You ask the three families living closest to you. If two of three say yes, you decide it is good. KNN is exactly that: it asks the k nearest known cases and goes with the majority.
There is no training in the usual sense. KNN is a "lazy learner": it just remembers the data and does the work at prediction time.
Example
A new customer. The five nearest known customers, closest first:
Neighbour distance class
A 1.0 Red
B 1.5 Blue
C 2.0 Red
D 2.2 Blue
E 3.0 Blue
k = 3 -> A, B, C -> Red 2, Blue 1 -> predict Red
k = 5 -> A..E -> Red 2, Blue 3 -> predict BlueThe answer flipped just by changing k. That is why k must be tuned with cross-validation.
from sklearn.neighbors import KNeighborsClassifier
model = KNeighborsClassifier(n_neighbors=5)
model.fit(X_train, y_train) # just stores the data
model.predict(X_test)Because it relies on distance, scale the features first.
What to say in the interview
"KNN finds the k closest training points to the query and votes, or averages for regression. It has no real training step, needs feature scaling, and k controls the bias-variance trade-off, chosen by cross-validation."
Common mistakes
- Forgetting to scale features.
- Choosing k on the test set.