Skip to main content
Terminal
Everything lives in config/evals.php:

Judge defaults

config/evals.php
Point the judge at a different model family than the agents under test — models grade their own family leniently. See The LLM Judge.

Gates and comparison

config/evals.php
Overridable per evaluation (gatePolicy()), per Pest eval (->gate(...)), and per CLI run (--min-score etc.). Pass-rate drops always count as regressions regardless of epsilon — see Baselines & Regressions.

Execution

config/evals.php
Concurrency parallelizes the agent invocations (the slow part); assertions, judges, and persistence always run in the parent process. Dry runs force sequential because SDK fakes live in process memory.

The price table

Cost tracking multiplies each sample’s token Usage against a user-maintained price table:
config/evals.php
USD per million tokens; cache_read/cache_write optional. Dated model ids reported by providers (gpt-5.4-2026-05-01) automatically match their family entry (gpt-5.4). Unknown models produce a null cost and a single warning — never an error, and never a silently-wrong number. Prices change; this table is yours to keep current.

Safety wordlist

config/evals.php