The lifecycle of a run
The pieces
Rows. A dataset yields rows: aninput (the prompt), optional prior messages for multi-turn, optional expected reference data, and free-form meta. Every row gets a content hash (input + messages + expected) so the same logical row is tracked across runs even when your file is reordered.
Samples. Agents are nondeterministic, so each row runs samples times. A row’s result is a pass rate and a score distribution (mean ± stddev) — never a single boolean.
Assertions first, judges last. Deterministic checks (substrings, tool calls, cost…) cost nothing and run first. An assertion marked ->gate() that fails hard-fails the sample — and skips its LLM judges entirely, so broken samples never spend judge tokens. Judges then score the surviving samples with persisted reasoning.
Persistence. Every sample writes its full context to your database: response text, structured output, tool calls, token usage, finish reason, duration, cost, and each assertion’s expected/actual/score — plus the judge’s reasoning. This is what powers the dashboard and everything below.
Baselines. In Pest, the first run that passes its gate becomes the suite’s baseline automatically. From the CLI it is explicit — pass --baseline (or promote a run later with evals:baseline or the dashboard). Later runs are joined to the baseline row-by-row (by content hash) and classified: regressed, improved, newly failing. See Baselines & Regressions.
The gate. Pass/fail is a run-level policy — minimum score, minimum pass rate, maximum regressions — applied after aggregation. In Pest, a failed gate fails the test with row-level detail; in the CLI, it sets the exit code.
Where things run
All three land in the same tables and the same dashboard.