Interview Q&A / AI & Generative AI

1. What is self-attention in a transformer?

   Self-attention lets every token in a sequence look at every other token and take a weighted mix of their information, so each word's representation reflects its context. The weights say how relevant each other token is.

   BeginnerQuick answer1 min read
2. Why did transformers replace RNNs for most language tasks?

   RNNs read tokens one at a time and squeeze history into a single state, which is slow to train and forgets distant words. Transformers process all tokens in parallel and connect any two directly through attention, so they train faster and use long context better.

   BeginnerQuick answer1 min read
3. What are queries, keys and values in attention?

   Each token is projected into three vectors. The query asks what this token is looking for, the key advertises what it contains, and the value is the information it passes on. Query-key matches set weights that mix the values.

   IntermediateQuick answer1 min read
4. What is a causal mask in a decoder-only model?

   A causal mask blocks each token from attending to later tokens, so position n sees only positions up to n. It matches generation, where future tokens do not exist yet, and lets training predict every position in parallel without cheating.

   IntermediateQuick answer1 min read
5. What is multi-head attention, and why use several heads?

   Multi-head attention runs several attention computations in parallel, each with its own learned projections, then combines the results. Different heads can capture different relationships, such as grammar, reference or nearby words, within the same layer.

   IntermediateQuick answer1 min read
6. What is the difference between encoder-only, decoder-only and encoder-decoder models?

   Encoder-only models like BERT read the whole input in both directions and suit classification and embeddings. Decoder-only models generate text left to right and power most chat LLMs. Encoder-decoder models read an input then generate an output, as in translation.

   IntermediateQuick answer1 min read
7. Why do transformers need positional encodings?

   Attention treats its input as an unordered set, so without position information "dog bites man" and "man bites dog" look alike. Positional encodings add order information to the token vectors so the model knows where each token sits.

   IntermediateQuick answer1 min read
8. Why does attention cost grow quickly as the input gets longer?

   Every token is compared with every other token, so the number of comparisons grows with the square of the sequence length. Doubling the input roughly quadruples the attention work and memory, which makes very long contexts expensive.

   IntermediateQuick answer1 min read