What is regression testing for prompts and models?
It means re-running a fixed evaluation set whenever you change a prompt, model, retrieval setting or tool, and comparing scores with the previous version. It catches cases where a change fixes one problem but quietly breaks others.
Explain it simply
You fix a bug in your app's login page and, without noticing, the payment page stops working. Regression tests exist to catch exactly that.
Prompts behave the same way. Adding "be concise" may improve summaries but cause the bot to drop the safety disclaimer. A provider updating a model version can change formats your parser relies on. Because outputs are not exact, you cannot check them with simple equality, so you use a mix: exact checks for formats and required fields, rule checks for forbidden content, and model or human grading for quality.
Practical routine: store the test set in version control, run it automatically on every change, show per-category scores, and block the release if a key metric drops. Pin model versions where possible and re-test before upgrading.
Example
Prompt v7 vs v8 on 120 cases
format valid: 100% -> 100%
correct answers: 88% -> 91%
refuses unsafe: 98% -> 90% <- regression, do not ship v8What to say in the interview
"Any prompt, model or retrieval change should trigger the full evaluation suite, compared with the previous version per category. I gate releases on key metrics and pin model versions so surprises are visible."