> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vizra.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Judge Calibration

> Measure how much your LLM judge agrees with humans — before you let it gate a release.

An LLM judge is a measurement instrument, and uncalibrated instruments lie confidently. Before gating CI on a judge, check its agreement with human labels.

## Label a dataset

Collect responses and score them yourself (or have your team do it):

```json storage/evals/labelled.jsonl theme={null}
{"input": "What is your refund policy?", "output": "Full refunds within 30 days with a receipt.", "human_score": 9}
{"input": "Do you price match?", "output": "Yes, we match any competitor!", "human_score": 2}
{"input": "Can I return opened items?", "output": "Opened items can be returned within 30 days.", "human_score": 7}
```

Each row needs an `output` (the response being graded) and a `human_score` (1–10) — or a `human_verdict` (`pass`/`fail`) if you'd rather label coarsely.

## Run the calibration

```bash Terminal theme={null}
php artisan evals:calibrate storage/evals/labelled.jsonl \
    --criteria="Answers using only documented store policy."
```

The judge scores every row against the same criteria you'd use in your evals, and you get:

* **Exact agreement** and **agreement within ±1** (or verdict accuracy for pass/fail labels)
* **Mean absolute error** in points
* The **worst disagreements**, each with the judge's reasoning — the fastest way to see *why* it diverges

```text Example output theme={null}
Rows with human scores ................................ 25
Exact agreement ....................................... 40.0%
Agreement within ±1 ................................... 84.0%
Mean absolute error ................................... 0.9 points

Worst disagreements ......................... judge vs human
  Do you price match?  .................................. 8 vs 2
    The response is confident and helpful in tone…
```

That disagreement is the tell: the judge rewarded confident tone over policy accuracy — so sharpen the criteria ("penalize any claim not in the documented policy") and calibrate again.

Use `--judge=App\Evals\Judges\PolicyJudge` to calibrate a [custom judge](/evals/judge#custom-judge-agents). Calibration runs are persisted like any other run, so you can track judge quality over time too.

<Tip>
  Within-±1 agreement above \~80% is a reasonable bar for gating CI with `min:` thresholds. Below that, fix the criteria or the judge model before trusting it — a judge you haven't calibrated is a random number generator with good grammar.
</Tip>