Assertions Reference

Deterministic assertions run inside ->assert(...), per sample, before any judge. The closure receives an assertion proxy, the row, and the full response:

Anatomy

->assert(fn ($a, $row, $response) => $a
    ->notEmpty()->gate()                 // ->gate(): failure hard-fails the sample, skips judges
    ->contains($row->expected())
    ->toolCalled('lookup_policy')->weight(2.0)   // ->weight(): share of the sample score
    ->costBelow(0.02))

Every assertion records its expected and actual values — failures come with receipts, in the test output and the dashboard. In class-based evaluations the same checks are $this->assert*() methods (e.g. $a->toolCalled(...) ⇢ $this->assertToolCalled(...)).

Content

Assertion	Checks
`contains($needle, $ignoreCase = true)`	Substring present
`notContains($needle)`	Substring absent
`containsAnyOf([...])` / `containsAllOf([...])`	At least one / all substrings
`startsWith($prefix)` / `endsWith($suffix)`	Response boundaries
`matchesRegex('/pattern/')`	Regex match
`lengthBetween($min, $max)` / `wordCountBetween($min, $max)`	Size bounds
`notEmpty()`	Non-blank response (a classic gate)
`isBritishSpelling()` / `isAmericanSpelling()`	Brand-voice spelling conventions

Structure

Assertion	Checks
`validJson()` / `jsonHasKey('a.b')`	Response parses as JSON / has a (dot-notation) key
`validXml()` / `xmlHasTag('tag')`	XML validity / tag presence
`outputHasKey('score')`	Structured-output agents: key exists
`outputKey('score', 5)` or `outputKey('score', fn ($v) => $v >= 1)`	Key equals a value, or passes a callback
`outputKeyMatches('id', '/^ORD-/')`	Key matches a regex

Agent behavior

These inspect the real AgentResponse — actual tool calls with their arguments, actual steps — not text parsing:

Assertion	Checks
`toolCalled('lookup_order')` / `toolNotCalled('escalate')`	Tool was / wasn’t invoked
`toolCalledWith('lookup_order', ['id' => 7])`	Called at least once with this argument subset (nested arrays supported, extra keys allowed)
`toolCallOrder(['lookup_order', 'create_return'])`	Tools appear in this relative order (others may interleave)
`stepsBelow(4)`	Agent loop finished in under N steps
`finishReason(FinishReason::Stop)`	How generation ended — failures flag `Length` as truncation and `ContentFilter` as refusal
`noPendingApprovals()`	Response isn’t paused on human-in-the-loop tool approval

Usage & cost

Assertion	Checks
`costBelow(0.02)`	Sample cost (USD, from the price table)
`tokensBelow(4000)`	Total tokens across prompt/completion/cache/reasoning
`cacheHitRateAbove(0.5)`	Prompt-cache effectiveness
`durationBelow(3000)`	Wall-clock milliseconds for the invocation
`modelUsed('gpt-5.4')` / `providerUsed('openai')`	Which model/provider actually served it

Safety pre-filters

Named honestly — these are wordlist and regex checks, useful as cheap gates, not classifiers:

Assertion	Checks
`containsNoBlockedWords([...])`	Built-in blocklist + `evals.safety.blocked_words` + extras
`noObviousPII()`	Regex-grade scan: emails, US SSNs/phones, card-shaped numbers, IPv4

Anything needing actual judgment — tone, toxicity in context, factuality — belongs to the LLM judge, not a wordlist.

Scalars

equals($expected, $actual), true($condition), false($condition), greaterThan($threshold, $actual), lessThan($threshold, $actual) — for checks you compute yourself from $row and $response.

Custom assertions

Implement the contract once, use it everywhere:

app/Evals/Assertions/MentionsOrderNumber.php

use Laravel\Ai\Responses\AgentResponse;
use Vizra\Evals\Assertions\Assertion;
use Vizra\Evals\Assertions\AssertionResult;
use Vizra\Evals\Dataset\Row;

class MentionsOrderNumber implements Assertion
{
    public function __invoke(AgentResponse $response, Row $row): AssertionResult
    {
        $found = preg_match('/ORD-\d{6}/', $response->text) === 1;

        return AssertionResult::bool(
            'mentions_order_number',
            $found,
            'an ORD-xxxxxx reference',
            $response->text,
            'Response never cites the order number.',
        );
    }
}

Using it

->assert(fn ($a) => $a->with(new MentionsOrderNumber))

Getting Started

Core Concepts

Writing Evals

Running & CI

Dashboard

Migration

Assertions Reference

Content

Structure

Agent behavior

Usage & cost

Safety pre-filters

Scalars

Custom assertions

​Content

​Structure

​Agent behavior

​Usage & cost

​Safety pre-filters

​Scalars

​Custom assertions

Content

Structure

Agent behavior

Usage & cost

Safety pre-filters

Scalars

Custom assertions