Skip to main content
An LLM judge is a measurement instrument, and uncalibrated instruments lie confidently. Before gating CI on a judge, check its agreement with human labels.

​
Label a dataset

Collect responses and score them yourself (or have your team do it):
storage/evals/labelled.jsonl
Each row needs an output (the response being graded) and a human_score (1–10) — or a human_verdict (pass/fail) if you’d rather label coarsely.

​
Run the calibration

Terminal
The judge scores every row against the same criteria you’d use in your evals, and you get:
  • Exact agreement and agreement within ±1 (or verdict accuracy for pass/fail labels)
  • Mean absolute error in points
  • The worst disagreements, each with the judge’s reasoning — the fastest way to see why it diverges
Example output
That disagreement is the tell: the judge rewarded confident tone over policy accuracy — so sharpen the criteria (“penalize any claim not in the documented policy”) and calibrate again. Use --judge=App\Evals\Judges\PolicyJudge to calibrate a custom judge. Calibration runs are persisted like any other run, so you can track judge quality over time too.
Within-±1 agreement above ~80% is a reasonable bar for gating CI with min: thresholds. Below that, fix the criteria or the judge model before trusting it — a judge you haven’t calibrated is a random number generator with good grammar.