Vizra Evals vs pest-plugin-evals

Pest 5 ships its own evals plugin (pestphp/pest-plugin-evals), and the obvious question is how the two relate. Short answer: different layers, happy coexistence — both are “just expectations,” they share the --evals flag, and they can live in the same file.

tests/Evals/SupportBotTest.php

use App\Agents\SupportBot;

// pest-plugin-evals: quick text-level checks, nothing stored
it('stays on topic', function () {
    expect(SupportBot::class)
        ->prompt('What is your refund policy?')
        ->toContain('30 days')
        ->toBeRelevant();
});

// vizra/evals: dataset-driven, sampled, judged — and recorded
it('handles the support corpus', function () {
    expect(SupportBot::class)->toPassEval(fn ($eval) => $eval
        ->dataset(base_path('evals/support.jsonl'))
        ->samples(3)
        ->assert(fn ($a, $row) => $a->toolCalled('lookup_policy')->contains($row->expected()))
        ->judge('Answers using only documented policy.', min: 7)
        ->gate(maxRegressions: 0)
    );
});

Both are skipped without --evals. This is worth understanding rather than trusting, because the two plugins genuinely fight over that flag: each one pops --evals out of the argument list, so only one of them ever sees it. They coexist because both set PEST_EVALS=1 in the environment before popping, and both read that variable in isEvalMode(). Whichever runs first enables the other. Verified with both installed side by side: without the flag both skip, with it both run.

pest-plugin-evals requires Pest 5 and pins laravel/ai to a narrow range — currently >=0.10.2 <0.11.0. Running the two together therefore means Laravel 13, and a laravel/ai upgrade may block one until the other catches up.

When to reach for which

Question	Use
”Does a sane answer come back for this one prompt?”	`pest-plugin-evals` — one line, ephemeral, perfect for smoke checks
”How does the agent do across my whole dataset?”	`toPassEval()` with a dataset
”Did this prompt change make anything worse?”	`toPassEval()` — baselines need recorded history
”Did it call the right tools with the right arguments?”	`toPassEval()` — assertions run against the real `AgentResponse` tool calls
”How does it behave mid-conversation / on production traffic?”	`toPassEval()` — multi-turn replay; pest-plugin-evals runs are single-turn
”What’s this costing per sample?”	`toPassEval()` — token usage and cost are recorded and assertable

The structural difference

pest-plugin-evals operates on response text: its scorers receive strings, its repeat(n) requires every sample to pass (a consistency gate), and results exist only in the terminal. That’s exactly right for its job — fast, ergonomic checks. Vizra Evals operates on the response object and time: sampled score distributions rather than all-or-nothing, assertions against real tool calls and usage, multi-turn context, and every run persisted — which is what makes baselines, regression gates, trend charts, and the dashboard possible.

A perfectly good setup: pest-plugin-evals expectations as cheap per-prompt smoke checks, and one toPassEval() per agent as the recorded, baseline-gated measurement. They’ll never conflict.

Getting Started

Core Concepts

Writing Evals

Running & CI

Dashboard

Migration

Vizra Evals vs pest-plugin-evals

When to reach for which

The structural difference

​When to reach for which

​The structural difference

When to reach for which

The structural difference