Fit a line to the points (1,2), (2,4), (3,5) and compute R squared.
Slope = 1.5, intercept = 0.667, so y = 0.667 + 1.5x. R squared = 1 - 0.1667/4.6667, about 0.964: the line explains about 96% of the variation in y.
Explain it simply
Least squares chooses the line that makes the total of squared vertical gaps (residuals) as small as possible. For one feature there are neat formulas: slope = Sxy / Sxx, intercept = mean of y minus slope times mean of x.
Example
x = 1, 2, 3 mean x = 2
y = 2, 4, 5 mean y = 11/3 = 3.667
Sxy = (1-2)(2-3.667) + (2-2)(4-3.667) + (3-2)(5-3.667)
= 1.667 + 0 + 1.333 = 3
Sxx = 1 + 0 + 1 = 2
slope = 3 / 2 = 1.5
intercept = 3.667 - 1.5 x 2 = 0.667Predictions: 2.167, 3.667, 5.167. Residuals: -0.167, 0.333, -0.167.
SS_res = 0.0278 + 0.1111 + 0.0278 = 0.1667
SS_tot = (2-3.667)^2 + (4-3.667)^2 + (5-3.667)^2
= 2.7778 + 0.1111 + 1.7778 = 4.6667
R^2 = 1 - 0.1667 / 4.6667 = 1 - 0.0357 = 0.964Predict for x = 4: 0.667 + 1.5 x 4 = 6.667.
What to say in the interview
"Least squares gives slope Sxy over Sxx and the line passes through the mean point. R squared is one minus residual sum of squares over total sum of squares, here about 0.96. On a real task I would check R squared on held-out data, since training R squared can only rise as I add features."
Common mistakes
- Using R squared on training data to compare models with different numbers of features.
- Reading a high R squared as proof of causation.