Writing Pest Evals

Evals live in your test suite. There’s no separate framework to learn: toPassEval() is an expectation like any other, and the recording, baselines, and dashboard happen behind it.

tests/Evals/SupportBotTest.php

use App\Agents\SupportBot;

it('answers support questions from documented policy', function () {
    expect(SupportBot::class)->toPassEval(fn ($eval) => $eval
        ->dataset(base_path('evals/support.jsonl'))
        ->samples(3)
        ->assert(fn ($a, $row) => $a
            ->notEmpty()->gate()
            ->contains($row->expected()))
        ->judge('Answers using only documented store policy.', min: 7)
        ->gate(minScore: 0.8, maxRegressions: 0)
    );
});

The subject

expect(...) accepts a Laravel AI agent class-string, an agent instance, or a Closure(Row $row): AgentResponse — the closure escape hatch lets you evaluate anything that can produce an AgentResponse.

The `$eval` builder

Method	What it does
`dataset($pathOrArrayOrDataset)`	`.jsonl`/`.csv` path, inline array of rows, or any `Dataset`
`fromConversations(take: 50)`	Build the dataset from the agent’s stored production conversations
`samples(int)`	Runs per row (default 1 — set it higher; see Scoring)
`assert(fn ($a, $row, $response))`	Deterministic checks per sample — see Assertions
`judge($criteria, min: 7, ...)`	LLM-judged criteria — see The Judge; repeatable
`gate(minScore:, minPassRate:, maxRegressions:)`	Run-level pass policy
`across([...])`	Provider/model matrix — each combo becomes its own series
`suite('name')`	Override the recording identity (defaults to the test’s description)
`using(SupportQuality::class)`	Run a full class-based evaluation instead of inline config
`concurrency(int)`	Parallel agent invocations for this run

Skip semantics

Evals call real models, so they never run by accident:

Terminal

./vendor/bin/pest             # all toPassEval tests are skipped
./vendor/bin/pest --evals     # they run
PEST_EVALS=1 ./vendor/bin/pest   # equivalent, for CI matrices

The --evals flag and PEST_EVALS variable are the same contract used by pestphp/pest-plugin-evals, so the two plugins coexist cleanly in one suite — whichever consumes the flag, both see eval mode. See Vizra Evals vs pest-plugin-evals.

What failure looks like

A failed gate fails the test with the receipts — aggregate score and pass rate, the run id (look it up in the dashboard), regressed rows with before → after scores, and the worst failing rows with sample counts. A run where every sample errored always fails, whatever the gate says.

Suites and recording

Each toPassEval test records under a suite named after the test (e.g. pest: answers support questions from documented policy) — that’s the identity baselines and history attach to. Rename the test and you start a fresh history, so pin it with ->suite('support-quality') if you expect descriptions to churn.

Group your evals (->group('evals') or a tests/Evals directory) so you can run them in a dedicated CI job with its own schedule and budget — see Running in CI.

Getting Started

Core Concepts

Writing Evals

Running & CI

Dashboard

Migration

Writing Pest Evals

The subject

The `$eval` builder

Skip semantics

What failure looks like

Suites and recording

​The subject

​The $eval builder

​Skip semantics

​What failure looks like

​Suites and recording

The subject

The `$eval` builder

Skip semantics

What failure looks like

Suites and recording