The LLM Judge

Some qualities can’t be asserted with string checks: accuracy against policy, tone, helpfulness. For those, an LLM judge scores each sample 1–10 against your criteria — through structured output, so there’s no fragile response parsing anywhere — and its reasoning is persisted with every score.

The basics

->judge('Answers using only documented store policy. Invents nothing.', min: 7)

The judge sees the row’s input (or full multi-turn transcript), the expected reference data when present, the agent’s response, and — since 0.3 — the instructions the agent under test was given. A sample passes the judge when its raw score meets min (default from evals.judge.min_score, 7). The normalized score (raw / 10) feeds the sample score.

Dimensions

Require per-dimension minimums on top of the overall score:

Multi-dimensional

->judge('Handles the support request well.', min: 7, dimensions: [
    'accuracy' => 7,
    'tone' => 6,
])

The judge returns a score per dimension; any dimension below its threshold fails the sample even if the overall score passes. Dimension scores are recorded too.

The judge reads the agent’s instructions

Since 0.3, the agent’s system prompt is passed to the judge automatically. Nothing to configure. It matters more than it sounds. Without it the judge grades prose against a rulebook it cannot read, so the agent’s own documented behaviour reads as invention:

agent: “You have 30 days from delivery to return an item, and it must be in its original condition” judge: “introduces multiple unstated policies”

Every fact in that answer was in the agent’s instructions. On an 11-row suite, the same agent and the same model scored 63.7% with a criteria that assumed the judge knew the policy and 96.2% once the policy was pasted into the criteria by hand. Passing the instructions is that workaround, done once and kept in step automatically. The instructions are resolved once per evaluation, not per sample. They are skipped for a Closure target, which has no instructions to read, and for an agent that will not resolve — a judge without context is a worse grade, not a failed run.

Judged scores move when you upgrade to 0.3. The judge now sees something it did not see before on every judge() call — generally scoring higher, where failures were the judge not knowing what the agent was permitted to do. Re-baseline before reading a comparison across the upgrade.

Some criteria are deliberately about the response alone — tone, reading age, format — where the system prompt would only bias the grade or spend tokens. In a class-based evaluation, turn it off per judge:

Inside evaluate()

$this->judge()
    ->criteria('Reads at a ninth-grade level.')
    ->withoutTargetInstructions()
    ->minScore(7);

The Pest ->judge(...) surface always includes them; drop to the class-based form for a criteria that needs them gone.

Judges run last — and gates protect your budget

Judges execute only after all deterministic assertions, and a failed ->gate() skips them entirely for that sample. An empty response never costs judge tokens. (Configurable via evals.judge.skip_on_gate_failure.)

Choosing the judge model

Per judge

->judge('...', min: 7, provider: 'anthropic', model: 'claude-sonnet-5')

Defaults come from evals.judge.provider / evals.judge.model (EVALS_JUDGE_PROVIDER / EVALS_JUDGE_MODEL).

Point the judge at a different model family than the agent under test. Models grade their own family’s output leniently; cross-family judging is the cheapest bias reduction you can buy.

Budget for the judge, not for the agent. The judge reads the criteria, the input, the response and the agent’s instructions, and it runs once per judged sample — so it routinely costs several times the thing it is grading. On one measured run, a gpt-5 judge grading a gpt-5-mini agent cost $0.1143 against the agent's$ 0.0080 — 14×. Nothing is wrong when you see that; it is what judging costs. Both figures are broken out in the run summary and in the dashboard, so pick the judge model with the total in view — a smaller judge on a well-calibrated criteria is often the difference between running a suite nightly and running it monthly.

Custom judge agents

The default judge is a structured-output agent returning {score, reasoning} (+ dimensions). Supply your own for domain-specific rubrics:

app/Evals/Judges/PolicyJudge.php

use Illuminate\Contracts\JsonSchema\JsonSchema;
use Laravel\Ai\Contracts\Agent;
use Laravel\Ai\Contracts\HasStructuredOutput;
use Laravel\Ai\Promptable;

class PolicyJudge implements Agent, HasStructuredOutput
{
    use Promptable;

    public function instructions(): string
    {
        return 'You are a strict compliance reviewer for Acme Store...';
    }

    public function schema(JsonSchema $schema): array
    {
        return [
            'score' => $schema->integer()->min(1)->max(10)->required(),
            'reasoning' => $schema->string()->required(),
        ];
    }
}

Using it

->judge('Complies with the returns policy.', min: 8, using: PolicyJudge::class)

Pairwise comparison

In class-based evaluations, judge one response against another — e.g. the reply your agent gave in production:

Inside evaluate()

$this->judge()
    ->comparedTo($row->expected())   // reference text, or another AgentResponse
    ->prefer('actual');              // fail unless the new response wins

Win scores 1.0, tie 0.5, loss 0.0. Only an outright win passes: prefer defaults to 'actual', so a tie fails the assertion whether or not you called ->prefer(). The 0.5 still counts toward the sample score — a draw is worth something, it just isn’t a pass.

Reasoning is the debugging payload

Every judgment stores its reasoning alongside the score. When a sample scores 6/10, the dashboard shows why in the judge’s own words — which is usually the fastest route to the prompt fix.

Don’t trust an uncalibrated judge. Measure its agreement with human labels before you gate CI on it — see Judge Calibration.

Getting Started

Core Concepts

Writing Evals

Running & CI

Dashboard

Migration

The LLM Judge

Dimensions

The judge reads the agent’s instructions

Judges run last — and gates protect your budget

Choosing the judge model

Custom judge agents

Pairwise comparison

Reasoning is the debugging payload

​Dimensions

​The judge reads the agent’s instructions

​Judges run last — and gates protect your budget

​Choosing the judge model

​Custom judge agents

​Pairwise comparison

​Reasoning is the debugging payload

Dimensions

The judge reads the agent’s instructions

Judges run last — and gates protect your budget

Choosing the judge model

Custom judge agents

Pairwise comparison

Reasoning is the debugging payload