Why did transformers replace RNNs for most language tasks?
RNNs read tokens one at a time and squeeze history into a single state, which is slow to train and forgets distant words. Transformers process all tokens in parallel and connect any two directly through attention, so they train faster and use long context better.
Explain it simply
An RNN is like passing a message down a line of people by whispering. By the tenth person the beginning is blurry, and nobody can start until the person before has finished.
Transformers put everyone in a room with a shared whiteboard. Every token can look directly at every other token. That fixes two problems. Training can run in parallel across the sequence, so GPUs are used well and much bigger models are practical. And information from the start of a passage does not need to survive a long chain of steps.
The trade-off is cost: attention compares all pairs of tokens, so work grows roughly with the square of the sequence length, which is why long inputs are expensive.
Example
RNN: t1 -> t2 -> t3 -> t4 (strictly sequential, one state carries all)
Transformer: t1 <-> t2 <-> t3 <-> t4 (all pairs visible, computed together)What to say in the interview
"Transformers removed the sequential bottleneck. Attention gives direct paths between any two tokens and allows parallel training, which scaled far better than RNNs and LSTMs. The cost is quadratic attention in sequence length."