What is the difference between offline and online evaluation of an LLM feature?
Offline evaluation tests a model or prompt on a fixed dataset before release, which is repeatable and safe. Online evaluation measures real behaviour after release, through user feedback, behavioural signals and A/B tests, and catches problems the dataset missed.
Explain it simply
A cricket team practises in the nets before a match, which is offline. The real match, with crowd and pressure, is online. Nets are safe and repeatable. Matches show what actually happens.
Offline evaluation runs your test set against each new prompt or model. It is fast and cheap and stops obvious regressions. But it only covers cases you thought of.
Online evaluation watches production: thumbs up and down, whether the user rephrased the question, how often they escalated to a human, task completion, latency and cost. You may A/B test two versions on a small share of traffic. Review a sample of real conversations regularly, and add newly found failures to the offline dataset so the two loops feed each other.
Example
Offline: 150-case test set -> pass rate 92% -> safe to try
Online: 5% of traffic on new prompt for a week
compare: resolution rate, escalation rate, thumbs-down rateWhat to say in the interview
"Offline evaluation gives repeatable pre-release checks on a fixed set; online evaluation reveals real-world behaviour via feedback and A/B tests. I use both and feed production failures back into the offline set."