Evals: how to know your model got worse
Golden sets, LLM-as-judge, and why "it feels better" is not a release gate.
Every prompt change is a deployment to a system with no unit tests unless you build them. An eval set is that test suite: a fixed collection of inputs with known-good outputs or scoring criteria, run on every change.
Start with fifty examples you curated by hand from real traffic, including the ugly ones. Fifty good examples beat five thousand synthetic ones, because the synthetic ones share the blind spots of the model that wrote them.
For scoring, use exact match or a rule wherever you can, and an LLM judge only for the genuinely subjective parts. Calibrate the judge against human ratings once, then keep the judge model and its prompt frozen — a moving judge makes your history meaningless.
The shape of it
- 01
Collect
Real inputs, including failures and edge cases.
- 02
Label
Expected output, or the criteria a good answer must meet.
- 03
Score
Deterministic rules first, LLM judge only where needed.
- 04
Gate
No prompt or model change ships if the score regresses.
Want this applied to your situation?
Sessions are direct and specific — you leave with a decision, not a reading list.
Book a session