LLM-as-a-judge is most useful when it makes a human evaluation process more repeatable. It is least useful when it hides an undefined standard behind a plausible score.
Three rules
- Define the rubric before choosing the judge. State the outcome, evidence, and failure modes that matter. A judge cannot compensate for an unclear quality bar.
- Keep representative human calibration cases. Compare judge output with reviewed examples, especially on ambiguous and safety-sensitive cases.
- Measure slices, not just averages. A system can look healthy overall while failing for a specific tool, task type, or user language.
The goal is not to automate judgement away. It is to make the feedback loop fast enough that people can spend their attention on the examples where it matters most.