> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vizra.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Running in CI

> Fail the build when your agent regresses — with cost kept firmly under control.

The whole design converges here: a pipeline step that runs your evals against real models and fails when quality drops below the baseline.

## The two shapes

**Pest-shaped** — evals live in your test suite, CI adds the flag:

```yaml .github/workflows/evals.yml theme={null}
name: Evals

on:
  pull_request:
    paths: ['app/Agents/**', 'app/Prompts/**', 'evals/**']
  schedule:
    - cron: '0 6 * * *'   # nightly drift check

jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: shivammathur/setup-php@v2
        with:
          php-version: '8.4'
      - run: composer install --no-interaction
      - run: php artisan migrate --force
      - run: ./vendor/bin/pest --evals tests/Evals
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
```

**CLI-shaped** — no Pest in the pipeline, JSON artifact out:

```yaml The eval step theme={null}
      - run: php artisan evals:run --compare=baseline --baseline --output=json > eval-report.json
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: eval-report, path: eval-report.json }
```

<Note>
  `--baseline` is doing real work here. Unlike the Pest surface, the CLI never promotes a
  baseline on its own — without it there is nothing for `--compare=baseline` to resolve, and
  every run reports *"No reference run found for --compare=baseline"* forever. With it, each
  gate-passing run becomes the reference for the next.
</Note>

Exit codes do the gating: `0` clean, `1` regression/gate failure, `2` harness failure. See [CLI](/evals/cli).

## Baselines in CI

Baselines are established differently on each surface — in Pest the first gate-passing run auto-baselines its suite; from the CLI you ask for it with `--baseline`. Either way, the CI question is *where the baseline lives*: comparison needs a database with history. Two workable setups:

* **Shared eval database** (recommended): point CI's `DB_*` at a small persistent database (the same one your [dashboard](/evals/dashboard) reads). Every CI run lands in history, baselines persist, and the dashboard shows your CI runs' trends for free.
* **Ephemeral DB, artifact-driven**: run with a throwaway database and treat the JSON report as the record. You lose `--compare` (no history to compare against) but keep absolute gates (`--min-score`, `--min-pass-rate`).

<Tip>
  [Vizra Cloud](/cloud) is the third option, and the reason it exists: CI reports its runs to
  a hosted history, so an ephemeral CI database costs you nothing and the same trend line
  covers your laptop, your branch and main. Set `VIZRA_CLOUD_KEY` as a CI secret.
</Tip>

<Warning>
  Trigger evals **deliberately**, not on every push: path filters (only when agents/prompts/datasets change), a nightly schedule, or a PR label. They cost real tokens and real minutes.
</Warning>

## Controlling cost

* `samples(2–3)` is usually enough signal for CI; save `samples(10)` for investigations.
* Gates skip judges on broken samples automatically — your worst PRs are your cheapest.
* Set `min:` thresholds only on [calibrated judges](/evals/calibration); a miscalibrated judge burns tokens *and* trust.
* The JSON report includes total and judge cost per run — chart it; eval spend that drifts up usually means a dataset quietly grew.

## Keys and safety rails

Provider keys come from CI secrets, judge configuration via `EVALS_JUDGE_PROVIDER` / `EVALS_JUDGE_MODEL`. And keep a wiring check in the *normal* test lane so breakage surfaces before the expensive lane runs:

```bash Cheap lane (every push) theme={null}
php artisan evals:run --dry-run   # every suite, zero tokens — proves datasets parse and suites boot
```