What is LLM-as-a-judge, and what biases should you watch for?
LLM-as-a-judge uses a model to grade another model's outputs against a rubric, scaling evaluation beyond human review. Known biases include preferring longer answers, favouring the first option shown, and favouring its own style, so calibrate against human labels.
Explain it simply
Reading 5,000 answers by hand is impractical, so teams ask a strong model to score them against a rubric, such as "is every claim supported by the context? yes or no".
It works reasonably well when the rubric is specific. It has known weaknesses. Judges often prefer longer, more elaborate answers even when a short one is better. When comparing two answers, they may favour whichever came first. They can favour text written in their own style, and they can be fooled by confident wrong statements.
Mitigations: write narrow, binary or small-scale criteria, randomise or swap answer order, ask for a brief justification, and most importantly compare the judge's scores with human labels on a sample to see how much you can trust it. Re-check when the judge model changes.
Example
Pairwise test, run twice with order swapped:
Run 1: A first, B second -> judge picks A
Run 2: B first, A second -> judge picks B
Inconsistent: order bias detected, so treat this pair as a tie.What to say in the interview
"LLM judges scale grading but carry position, length and self-preference biases. I use focused rubrics, swap orders, and validate the judge against human-labelled samples before trusting its scores."