What are queries, keys and values in attention?
Each token is projected into three vectors. The query asks what this token is looking for, the key advertises what it contains, and the value is the information it passes on. Query-key matches set weights that mix the values.
Explain it simply
Think of a library search. Your query is "books on cricket statistics". Each book has a label, its key, describing its subject, and the actual content, its value. You compare your query with every label, find the best matches, and read mostly from those books.
In attention, every token produces all three by multiplying its embedding with three learned matrices. A token's query is compared with every token's key using a dot product. The scores are scaled, passed through softmax to become weights, and used to average the value vectors.
Keeping keys and values separate lets a token be easy to find for one reason while handing over different information. These are learned during training, not written by hand.
Example
# single-head attention for one token (conceptual)
scores = [q @ k_j for k_j in keys] # match query with each key
weights = softmax(scores) # sum to 1
output = sum(w * v_j for w, v_j in zip(weights, values))The output is a blend of values, weighted by how well each key matched the query.
What to say in the interview
"Q, K and V are learned linear projections of each token. The dot product of a query with all keys, scaled and softmaxed, gives attention weights, and the output is the weighted sum of the values. Separating them lets the model decide what to look for, what to advertise and what to pass along."
Common mistakes
- Saying keys and values are the same thing.
- Forgetting the scaling and softmax step.