What is self-attention in a transformer?
Self-attention lets every token in a sequence look at every other token and take a weighted mix of their information, so each word's representation reflects its context. The weights say how relevant each other token is.
Explain it simply
In the sentence "Ravi gave Amit his notes because he was absent", what does "he" mean? To decide, you glance back at the other words and weigh them. Self-attention does exactly that for every word at once.
For each token the model computes a relevance score against all tokens in the sequence, turns the scores into weights that add to 1, and builds a new vector for that token as a weighted blend of the others. The word "bank" near "river" ends up with a different vector than "bank" near "loan".
Because all tokens are handled together rather than one after another, the work can be done in parallel on GPUs, and a token can connect to a distant one in a single step.
Example
Sentence: "The train was late because it was foggy"
Attention weights for the token "it" (illustrative):
train 0.62 late 0.10 foggy 0.15 the 0.01 ...
The vector for "it" is updated mostly with information from "train".What to say in the interview
"Self-attention computes, for each token, weights over all tokens in the sequence and produces a context-aware representation as the weighted sum. It lets the model resolve references and long-range dependencies and runs in parallel, unlike recurrent models."
Common mistakes
- Describing attention as the model "focusing" like a human. It is a learned weighted average.