What is log loss (cross-entropy), and why does logistic regression not use mean squared error?
Log loss is -ln(p) for the probability given to the true class. It punishes confident wrong predictions very heavily. With a sigmoid, squared error gives a non-convex loss and weak gradients; log loss is convex and gives clean gradients.
Explain it simply
Imagine a weather forecaster who says "99% no rain" and it pours. That is worse than one who said "60% no rain". Log loss captures this idea: being confidently wrong costs a lot, being unsure and wrong costs less, being right with confidence costs almost nothing.
Example
For one example with true label 1, loss = -ln(p) where p is the predicted probability of class 1.
p = 0.9 -> -ln(0.9) = 0.105 (confident and right: tiny loss)
p = 0.6 -> -ln(0.6) = 0.511
p = 0.1 -> -ln(0.1) = 2.303 (confident and wrong: large loss)
p -> 0 -> loss grows without limitFor label 0 the loss is -ln(1 - p). Averaging over all rows gives the model's log loss.
Why not squared error? With p = 0.1 and y = 1, squared error is (0.9)^2 = 0.81, a bounded penalty, and the sigmoid's flat tail makes the gradient tiny, so learning stalls. Combined with the sigmoid, log loss gives a gradient of (p - y) times x, which is simple and large when the model is wrong. It is also convex, so gradient descent has no local minima.
What to say in the interview
"Log loss is the average of -ln of the probability assigned to the true class. It is the negative log likelihood of the data under the model, it is convex for logistic regression, and it penalises overconfident mistakes heavily. Squared error with a sigmoid is non-convex and learns slowly when the model is badly wrong."
Common mistakes
- Mixing up log base: scikit-learn uses the natural log.
- Thinking lower log loss always means higher accuracy; it measures probability quality.