> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vizra.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# The LLM Judge

> Structured-output scoring with reasoning you keep — no regex parsing, no black boxes.

Some qualities can't be asserted with string checks: accuracy against policy, tone, helpfulness. For those, an LLM judge scores each sample 1–10 against your criteria — through **structured output**, so there's no fragile response parsing anywhere — and its reasoning is persisted with every score.

```php The basics theme={null}
->judge('Answers using only documented store policy. Invents nothing.', min: 7)
```

The judge sees the row's input (or full multi-turn transcript), the `expected` reference data when present, the agent's response, and — since 0.3 — [the instructions the agent under test was given](#the-judge-reads-the-agents-instructions). A sample passes the judge when its raw score meets `min` (default from `evals.judge.min_score`, 7). The normalized score (`raw / 10`) feeds the [sample score](/evals/scoring).

## Dimensions

Require per-dimension minimums on top of the overall score:

```php Multi-dimensional theme={null}
->judge('Handles the support request well.', min: 7, dimensions: [
    'accuracy' => 7,
    'tone' => 6,
])
```

The judge returns a score per dimension; any dimension below its threshold fails the sample even if the overall score passes. Dimension scores are recorded too.

## The judge reads the agent's instructions

Since **0.3**, the agent's system prompt is passed to the judge automatically. Nothing to configure.

It matters more than it sounds. Without it the judge grades prose against a rulebook it cannot read, so the agent's own documented behaviour reads as invention:

> *agent:* "You have 30 days from delivery to return an item, and it must be in its original condition"
> *judge:* "introduces multiple unstated policies"

Every fact in that answer was in the agent's instructions. On an 11-row suite, the same agent and the same model scored **63.7%** with a criteria that assumed the judge knew the policy and **96.2%** once the policy was pasted into the criteria by hand. Passing the instructions is that workaround, done once and kept in step automatically.

The instructions are resolved **once per evaluation**, not per sample. They are skipped for a `Closure` target, which has no instructions to read, and for an agent that will not resolve — a judge without context is a worse grade, not a failed run.

<Warning>
  **Judged scores move when you upgrade to 0.3.** The judge now sees something it did not see before on every `judge()` call — generally scoring higher, where failures were the judge not knowing what the agent was permitted to do. Re-baseline before reading a comparison across the upgrade.
</Warning>

Some criteria are deliberately about the response alone — tone, reading age, format — where the system prompt would only bias the grade or spend tokens. In a [class-based evaluation](/evals/class-based-evaluations), turn it off per judge:

```php Inside evaluate() theme={null}
$this->judge()
    ->criteria('Reads at a ninth-grade level.')
    ->withoutTargetInstructions()
    ->minScore(7);
```

The Pest `->judge(...)` surface always includes them; drop to the class-based form for a criteria that needs them gone.

## Judges run last — and gates protect your budget

Judges execute only after all deterministic assertions, and **a failed `->gate()` skips them entirely** for that sample. An empty response never costs judge tokens. (Configurable via `evals.judge.skip_on_gate_failure`.)

## Choosing the judge model

```php Per judge theme={null}
->judge('...', min: 7, provider: 'anthropic', model: 'claude-sonnet-5')
```

Defaults come from `evals.judge.provider` / `evals.judge.model` (`EVALS_JUDGE_PROVIDER` / `EVALS_JUDGE_MODEL`).

<Warning>
  **Point the judge at a different model family than the agent under test.** Models grade their own family's output leniently; cross-family judging is the cheapest bias reduction you can buy.
</Warning>

**Budget for the judge, not for the agent.** The judge reads the criteria, the input, the response and the agent's instructions, and it runs once per judged sample — so it routinely costs several times the thing it is grading. On one measured run, a `gpt-5` judge grading a `gpt-5-mini` agent cost **$0.1143 against the agent's $0.0080** — 14×. Nothing is wrong when you see that; it is what judging costs. Both figures are broken out in the run summary and in the [dashboard](/evals/dashboard), so pick the judge model with the total in view — a smaller judge on a well-calibrated criteria is often the difference between running a suite nightly and running it monthly.

## Custom judge agents

The default judge is a structured-output agent returning `{score, reasoning}` (+ dimensions). Supply your own for domain-specific rubrics:

```php app/Evals/Judges/PolicyJudge.php theme={null}
use Illuminate\Contracts\JsonSchema\JsonSchema;
use Laravel\Ai\Contracts\Agent;
use Laravel\Ai\Contracts\HasStructuredOutput;
use Laravel\Ai\Promptable;

class PolicyJudge implements Agent, HasStructuredOutput
{
    use Promptable;

    public function instructions(): string
    {
        return 'You are a strict compliance reviewer for Acme Store...';
    }

    public function schema(JsonSchema $schema): array
    {
        return [
            'score' => $schema->integer()->min(1)->max(10)->required(),
            'reasoning' => $schema->string()->required(),
        ];
    }
}
```

```php Using it theme={null}
->judge('Complies with the returns policy.', min: 8, using: PolicyJudge::class)
```

## Pairwise comparison

In [class-based evaluations](/evals/class-based-evaluations), judge one response *against another* — e.g. the reply your agent gave in production:

```php Inside evaluate() theme={null}
$this->judge()
    ->comparedTo($row->expected())   // reference text, or another AgentResponse
    ->prefer('actual');              // fail unless the new response wins
```

Win scores `1.0`, tie `0.5`, loss `0.0`. Only an outright win passes: `prefer` defaults to `'actual'`, so a tie fails the assertion whether or not you called `->prefer()`. The `0.5` still counts toward the sample score — a draw is worth something, it just isn't a pass.

## Reasoning is the debugging payload

Every judgment stores its reasoning alongside the score. When a sample scores 6/10, the [dashboard](/evals/dashboard) shows *why* in the judge's own words — which is usually the fastest route to the prompt fix.

<Tip>
  Don't trust an uncalibrated judge. Measure its agreement with human labels before you gate CI on it — see [Judge Calibration](/evals/calibration).
</Tip>