Interview Q&A / AI & Generative AI

1. How do you measure the quality of retrieval in a RAG system?

   Use a set of questions with known correct passages and compute recall at k, the share of questions where a correct chunk appears in the top k, and MRR, which rewards ranking the correct chunk higher. Check retrieval separately from the generated answer.

   IntermediateQuick answer1 min read
2. In RAG evaluation, what is the difference between faithfulness and answer relevance?

   Faithfulness asks whether every claim in the answer is supported by the retrieved context. Answer relevance asks whether the answer actually addresses the user's question. An answer can be faithful but off-topic, or on-topic but unsupported.

   IntermediateQuick answer1 min read
3. What is LLM-as-a-judge, and what biases should you watch for?

   LLM-as-a-judge uses a model to grade another model's outputs against a rubric, scaling evaluation beyond human review. Known biases include preferring longer answers, favouring the first option shown, and favouring its own style, so calibrate against human labels.

   IntermediateQuick answer1 min read
4. Why are BLEU and ROUGE poor metrics for judging LLM answers?

   They measure word overlap with a reference text. A correct answer worded differently scores low, while a fluent wrong answer sharing many words can score high. They suit narrow tasks like translation drafts, but not open-ended answer quality.

   IntermediateQuick answer1 min read