Interview Q&A / AI & Generative AI

1. What are queries, keys and values in attention?

   Each token is projected into three vectors. The query asks what this token is looking for, the key advertises what it contains, and the value is the information it passes on. Query-key matches set weights that mix the values.

   IntermediateQuick answer1 min read
2. What is multi-head attention, and why use several heads?

   Multi-head attention runs several attention computations in parallel, each with its own learned projections, then combines the results. Different heads can capture different relationships, such as grammar, reference or nearby words, within the same layer.

   IntermediateQuick answer1 min read
3. Why do transformers need positional encodings?

   Attention treats its input as an unordered set, so without position information "dog bites man" and "man bites dog" look alike. Positional encodings add order information to the token vectors so the model knows where each token sits.

   IntermediateQuick answer1 min read
4. Why does attention cost grow quickly as the input gets longer?

   Every token is compared with every other token, so the number of comparisons grows with the square of the sequence length. Doubling the input roughly quadruples the attention work and memory, which makes very long contexts expensive.

   IntermediateQuick answer1 min read