Running in CI

The whole design converges here: a pipeline step that runs your evals against real models and fails when quality drops below the baseline.

The two shapes

Pest-shaped — evals live in your test suite, CI adds the flag:

.github/workflows/evals.yml

name: Evals

on:
  pull_request:
    paths: ['app/Agents/**', 'app/Prompts/**', 'evals/**']
  schedule:
    - cron: '0 6 * * *'   # nightly drift check

jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: shivammathur/setup-php@v2
        with:
          php-version: '8.4'
      - run: composer install --no-interaction
      - run: php artisan migrate --force
      - run: ./vendor/bin/pest --evals tests/Evals
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

CLI-shaped — no Pest in the pipeline, JSON artifact out:

The eval step

      - run: php artisan evals:run --compare=baseline --baseline --output=json > eval-report.json
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: eval-report, path: eval-report.json }

--baseline is doing real work here. Unlike the Pest surface, the CLI never promotes a baseline on its own — without it there is nothing for --compare=baseline to resolve, and every run reports “No reference run found for —compare=baseline” forever. With it, each gate-passing run becomes the reference for the next.

Exit codes do the gating: 0 clean, 1 regression/gate failure, 2 harness failure. See CLI.

Baselines in CI

Baselines are established differently on each surface — in Pest the first gate-passing run auto-baselines its suite; from the CLI you ask for it with --baseline. Either way, the CI question is where the baseline lives: comparison needs a database with history. Two workable setups:

Shared eval database (recommended): point CI’s DB_* at a small persistent database (the same one your dashboard reads). Every CI run lands in history, baselines persist, and the dashboard shows your CI runs’ trends for free.
Ephemeral DB, artifact-driven: run with a throwaway database and treat the JSON report as the record. You lose --compare (no history to compare against) but keep absolute gates (--min-score, --min-pass-rate).

Vizra Cloud is the third option, and the reason it exists: CI reports its runs to a hosted history, so an ephemeral CI database costs you nothing and the same trend line covers your laptop, your branch and main. Set VIZRA_CLOUD_KEY as a CI secret.

Trigger evals deliberately, not on every push: path filters (only when agents/prompts/datasets change), a nightly schedule, or a PR label. They cost real tokens and real minutes.

Controlling cost

samples(2–3) is usually enough signal for CI; save samples(10) for investigations.
Gates skip judges on broken samples automatically — your worst PRs are your cheapest.
Set min: thresholds only on calibrated judges; a miscalibrated judge burns tokens and trust.
The JSON report includes total and judge cost per run — chart it; eval spend that drifts up usually means a dataset quietly grew.

Keys and safety rails

Provider keys come from CI secrets, judge configuration via EVALS_JUDGE_PROVIDER / EVALS_JUDGE_MODEL. And keep a wiring check in the normal test lane so breakage surfaces before the expensive lane runs:

Cheap lane (every push)

php artisan evals:run --dry-run   # every suite, zero tokens — proves datasets parse and suites boot

Getting Started

Core Concepts

Writing Evals

Running & CI

Dashboard

Migration

Running in CI

The two shapes

Baselines in CI

Controlling cost

Keys and safety rails

​The two shapes

​Baselines in CI

​Controlling cost

​Keys and safety rails

The two shapes

Baselines in CI

Controlling cost

Keys and safety rails