What is gradient descent, and what happens if the learning rate is too high or too low?
Gradient descent trains a model by repeatedly nudging its weights in the direction that reduces the loss fastest, which is opposite to the gradient. The learning rate sets the step size. Too small and training is very slow; too large and it overshoots the minimum and can bounce around or diverge.
Explain it simply
Imagine you are on a hill in thick fog and want to reach the lowest point of the valley. You cannot see the valley, but you can feel the slope under your feet. So you take a step downhill, feel the slope again, and take another step. Repeat until the ground is flat.
- The height of the ground is the loss: how wrong the model is.
- The slope under your feet is the gradient.
- The size of each step is the learning rate.
If your steps are tiny, you will get there, but it may take all day. If your steps are huge, you may jump right over the valley to the other hillside, and then jump back, never settling.
Example
Take a simple loss with one weight: L(w) = (w − 3)². The best value is w = 3. The gradient is 2(w − 3). Start at w = 0. The update rule is:
w_new = w − learning_rate × gradient
def train(lr, steps=3, w=0.0):
for _ in range(steps):
grad = 2 * (w - 3)
w = w - lr * grad
print(round(w, 3))
train(0.1) # 0.6, 1.08, 1.464 -> moving steadily towards 3
train(1.1) # 6.6, -1.32, 8.184 -> overshooting, getting worseWith learning rate 0.1, each step closes 20% of the remaining gap to 3. With 1.1, each step jumps past 3 and lands 1.2 times further away than before, so the loss grows. With 1.0, w jumps between 0 and 6 forever.
How it works
- Start with random (or zero) weights.
- Compute predictions and the loss on some training data.
- Compute the gradient of the loss with respect to every weight. In neural networks this is done with backpropagation.
- Update every weight: w = w − η × ∂L/∂w, where η is the learning rate.
- Repeat until the loss stops improving.
Three variants, based on how much data each step uses:
| Variant | Data per step | Trade-off |
|---|---|---|
| Batch | Whole dataset | Smooth, accurate steps; slow and memory-heavy on large data |
| Stochastic (SGD) | One example | Fast, noisy steps; the noise can help escape poor spots |
| Mini-batch | A small batch, e.g. 32–256 | The standard choice; works well on GPUs |
One full pass through the training data is called an epoch.
Real-world use
- Almost every neural network, from image classifiers to large language models, is trained with a variant of gradient descent, usually Adam or AdamW.
- Logistic regression on large datasets is often trained with SGD.
- Practitioners use learning rate schedules: a short warm-up, then gradually decreasing the rate so the model takes big steps early and fine steps near the end.
Common mistakes
- Not scaling features. If one feature is in lakhs and another is between 0 and 1, the loss surface is a long narrow valley, and gradient descent zig-zags slowly.
- Reading a loss that shoots up to NaN as a bug in data, when the learning rate is simply too high.
- Assuming gradient descent always finds the global minimum. For convex losses like linear and logistic regression it does (with a suitable learning rate); for neural networks it finds a good enough low point.
- Confusing an epoch with an iteration. An iteration is one update; an epoch is one pass over all the data.
What to say in the interview
"Gradient descent minimises the loss by repeatedly computing the gradient of the loss with respect to the weights and moving the weights a small step in the opposite direction. The learning rate controls the step size. If it is too low, training is slow and may stall; if it is too high, the updates overshoot the minimum, the loss oscillates or even diverges. In practice we use mini-batch gradient descent, often with an adaptive optimiser like Adam, scale the features, and use a learning rate schedule. I watch the training and validation loss curves to tune it."
Follow-up questions
- What is the difference between batch, stochastic and mini-batch gradient descent?
- What is momentum, and how does Adam differ from plain SGD?
- What are vanishing and exploding gradients?
- How would you choose a learning rate?
Go deeper
Read these once the basics above feel easy.
Momentum keeps a running average of past gradients and moves in that direction, which smooths zig-zagging and speeds up progress along shallow valleys. Adam combines momentum with a per-weight step size based on the recent size of that weight's gradients, so rarely updated weights get relatively larger steps. A quick way to pick a learning rate is a learning rate range test: increase the rate exponentially over a few hundred batches and choose a value a little below where the loss starts to rise.