What is multi-head attention, and why use several heads?
Multi-head attention runs several attention computations in parallel, each with its own learned projections, then combines the results. Different heads can capture different relationships, such as grammar, reference or nearby words, within the same layer.
Explain it simply
When you read a sentence you track several things at once: who did what, which pronoun points to whom, what tense is used. One attention pattern per layer would force all of that into a single set of weights.
Multi-head attention splits the work. The embedding is projected into several smaller subspaces, each head runs its own attention, and the head outputs are joined and mixed by another learned matrix. Each head is free to specialise, for example one may link verbs to subjects while another links a token to its neighbours.
Heads are not designed by humans. Researchers observe that some become interpretable and some look redundant. The total compute stays similar because each head works in a smaller dimension.
Example
Layer with 8 heads, model dimension 512:
each head works in 64 dimensions (512 / 8)
8 attention outputs -> concatenated back to 512 -> linear mixWhat to say in the interview
"Multiple heads let a layer attend to different kinds of relationships in parallel, each in a lower-dimensional subspace, and the outputs are concatenated and projected. It adds representational variety without much extra compute."
Common mistakes
- Claiming every head clearly learns a distinct human-readable role.