Baselines & Regressions

The core workflow: pick a run you trust, make it the baseline, and every future run is diffed against it. This is what turns evals from “did it pass?” into “is it getting worse?”

Baselines

Automatic: in Pest, the first run that passes its gate becomes its suite’s baseline. Nothing to configure.
Manual: promote any completed run from the dashboard, or:

Terminal

php artisan evals:baseline 01kyvrjyxp9ex0f3gtvznmafdn

One baseline per suite; promoting a new one demotes the old one transactionally.

How runs are joined

Rows are matched across runs by content hash — a fingerprint of the row’s input, prior messages, and expected data, computed when the dataset is read. Reordering your JSONL file, adding rows, or renaming files changes nothing; the same logical row lines up across months of runs. (For single-model runs the reported provider/model id is deliberately ignored in the join — providers rotate dated model ids like gpt-5-mini-2025-08-07, and that must not sever your history.)

Classification

For each row present in both runs:

Class	Condition
Regressed	Pass rate dropped at all, or score mean dropped by more than `evals.compare.epsilon` (default `0.05`)
Newly failing	Was passing every sample on the baseline; isn’t now (a subset of regressed)
Improved	The symmetric opposite
Unchanged	Everything else — score jitter within epsilon is noise, not signal

Rows only in the current run are new; rows only in the baseline are removed.

Why epsilon? You’re sampling a nondeterministic system — small score wobble is expected. A pass-rate drop is always a regression, but a 0.93 → 0.90 score drift shouldn’t page anyone. Tune evals.compare.epsilon to your tolerance.

Failing the build on regressions

In Pest

->gate(maxRegressions: 0)

What a regression failure looks like

Gate failed: 2 rows regressed against the reference run (allowed: 0).
  ↓ regressed: "What is your refund policy?" 96.7% → 51.7%
  ↓ regressed: "Can I return it?" 93.3% → 55.0%

Or without Pest:

Terminal

php artisan evals:run SupportQuality --compare=baseline --output=json
# exit 0 = clean · exit 1 = gate failed or regressions exceeded the allowance

--compare also accepts a run id or latest. The dashboard’s compare view renders the same diff — regressed and improved tables with before → after scores. See Running in CI for the full pipeline recipe.

Getting Started

Core Concepts

Writing Evals

Running & CI

Dashboard

Migration

Baselines & Regressions

Baselines

How runs are joined

Classification

Failing the build on regressions

​Baselines

​How runs are joined

​Classification

​Failing the build on regressions

Baselines

How runs are joined

Classification

Failing the build on regressions