Interview Q&A / AI & Generative AI

1. How would you evaluate an LLM application before launching it?

   Build a test set of realistic inputs with expected outcomes, define metrics tied to the real goal, run automated checks plus human review on a sample, include edge cases and adversarial inputs, set pass thresholds, and keep the set to catch regressions later.

   BeginnerQuick answer1 min read
2. What is a golden dataset in LLM evaluation?

   A golden dataset is a curated set of inputs paired with verified expected answers or grading criteria, reviewed by people. It is the fixed benchmark you re-run after every change to see whether quality improved or regressed on your real use case.

   BeginnerQuick answer1 min read
3. What is regression testing for prompts and models?

   It means re-running a fixed evaluation set whenever you change a prompt, model, retrieval setting or tool, and comparing scores with the previous version. It catches cases where a change fixes one problem but quietly breaks others.

   BeginnerQuick answer1 min read
4. What is the difference between offline and online evaluation of an LLM feature?

   Offline evaluation tests a model or prompt on a fixed dataset before release, which is repeatable and safe. Online evaluation measures real behaviour after release, through user feedback, behavioural signals and A/B tests, and catches problems the dataset missed.

   BeginnerQuick answer1 min read
5. How do you measure the quality of retrieval in a RAG system?

   Use a set of questions with known correct passages and compute recall at k, the share of questions where a correct chunk appears in the top k, and MRR, which rewards ranking the correct chunk higher. Check retrieval separately from the generated answer.

   IntermediateQuick answer1 min read
6. In RAG evaluation, what is the difference between faithfulness and answer relevance?

   Faithfulness asks whether every claim in the answer is supported by the retrieved context. Answer relevance asks whether the answer actually addresses the user's question. An answer can be faithful but off-topic, or on-topic but unsupported.

   IntermediateQuick answer1 min read
7. What is LLM-as-a-judge, and what biases should you watch for?

   LLM-as-a-judge uses a model to grade another model's outputs against a rubric, scaling evaluation beyond human review. Known biases include preferring longer answers, favouring the first option shown, and favouring its own style, so calibrate against human labels.

   IntermediateQuick answer1 min read
8. Why are BLEU and ROUGE poor metrics for judging LLM answers?

   They measure word overlap with a reference text. A correct answer worded differently scores low, while a fluent wrong answer sharing many words can score high. They suit narrow tasks like translation drafts, but not open-ended answer quality.

   IntermediateQuick answer1 min read