CLI Commands

Everything the Pest expectation does is also an artisan command — same engine, same tables, same dashboard.

`evals:run`

Terminal

php artisan evals:run SupportQuality              # one class-based suite
php artisan evals:run                              # every discovered suite (app/Evals)
php artisan evals:run --filter=Support             # subset by name

Option	Effect
`--samples=5`	Override samples per row
`--dataset=path.jsonl`	Run a JSONL file instead of the dataset the evaluation declares — the same suite against different data
`--concurrency=1`	Sequential execution (debugging); default from config
`--dry-run`	Execute on SDK fakes — zero tokens, validates wiring (details)
`--compare=baseline`	Diff against the baseline (also accepts a run id or `latest`)
`--baseline`	Promote this run to baseline if its gate passes
`--output=json`	Machine-readable versioned document on stdout (progress on stderr)
`--report` / `--no-report`	Force or suppress pushing this run to Vizra Cloud, overriding the ambient configuration
`--min-score`, `--min-pass-rate`, `--max-regressions`	Gate overrides

Unlike Pest, the CLI never promotes a baseline on its own. The first run of a suite has nothing to compare against until you pass --baseline (or promote a run afterwards with evals:baseline) — see Baselines & Regressions.

Exit codes — the CI contract:

Code	Meaning
`0`	Gate passed
`1`	Gate failed, or regressions exceeded the allowance
`2`	Harness failure (unknown suite, unreadable dataset, run crashed)

The terminal output is a per-row table — pass counts, score ± spread, cost — plus totals; with --compare, regressed/improved tables with before → after scores.

`evals:baseline`

Terminal

php artisan evals:baseline 01kyvrjyxp9ex0f3gtvznmafdn

Promotes any completed run to its suite’s baseline (transactionally demoting the previous one). See Baselines & Regressions.

`evals:calibrate`

Terminal

php artisan evals:calibrate storage/evals/labelled.jsonl --criteria="..."

Judge-vs-human agreement measurement — see Judge Calibration.

`make:eval`

Terminal

php artisan make:eval SupportQuality

Scaffolds a class-based evaluation in app/Evals plus a starter JSONL dataset in evals/data/.

JSON output

--output=json emits a stable, versioned document: run metadata (git sha/branch, config), aggregate summary, every row with its samples and assertion results, the comparison diff, and the gate verdict. Pipe it to artifacts storage, parse it in CI annotations, chart it — the schema is designed to be depended on.

CI one-liner

php artisan evals:run SupportQuality --compare=baseline --output=json > eval-report.json

Getting Started

Core Concepts

Writing Evals

Running & CI

Dashboard

Migration

CLI Commands

`evals:run`

`evals:baseline`

`evals:calibrate`

`make:eval`

JSON output

​evals:run

​evals:baseline

​evals:calibrate

​make:eval

​JSON output

`evals:run`

`evals:baseline`

`evals:calibrate`

`make:eval`

JSON output