> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vizra.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Scoring Model

> How samples become scores, scores become rates, and rates become a pass or fail.

Scores are the primary output; pass/fail is a policy applied on top. Here's the exact math, bottom to top.

## Assertion scores

* **Deterministic assertions** score `1.0` (passed) or `0.0` (failed).
* **Judge assertions** score `raw / 10`, normalized to `0–1`. Pairwise comparisons score win `1.0` / tie `0.5` / loss `0.0`.
* **Skipped or errored** assertions have no score (`null`) and are excluded from means.
* Every assertion can carry a `->weight(float)` (default `1.0`).

## Sample score

A sample is one invocation of the agent for one row.

* **If any gated assertion failed (or errored): the sample scores `0.0` and fails.** Gates are binary preconditions — they're *excluded* from the score mean, because "the response wasn't empty" isn't quality signal.
* Otherwise: `score = Σ(weightᵢ × scoreᵢ) / Σ(weightᵢ)` over non-gate assertions with scores.
* A sample whose only assertions were passing gates scores `1.0`.
* A sample whose agent invocation or evaluation threw is an **error**: no score, counted separately.

```php Weighting example theme={null}
$a->contains($row->expected())          // weight 1.0
  ->toolCalled('lookup_policy')->weight(2.0);   // counts double
// judge at min: 7 scoring 8/10 → 0.8, weight 1.0
// sample score = (1.0×1 + 1.0×2 + 0.8×1) / 4 = 0.95
```

## Row result

Each row (per model combo) aggregates its samples:

* `pass_rate` = passed samples ÷ all samples (errors count in the denominator)
* `score_mean`, `score_stddev` = over samples with scores (errors excluded from the mean)

The stddev matters: a row at `90% ± 2` and a row at `90% ± 25` are very different agents.

## Run result

Rows aggregate with equal weight: run `score_mean`/`score_stddev` over row means, run `pass_rate` as the mean of row pass rates — plus totals (rows, samples, errors, cost, judge cost, duration). All of it is persisted on the run's summary; nothing is recomputed later.

## The gate

```php In Pest theme={null}
->gate(minScore: 0.8, minPassRate: 0.9, maxRegressions: 0)
```

* `minScore` / `minPassRate` — thresholds on the run aggregates (`0–1`).
* `maxRegressions` — enforced against the suite's [baseline](/evals/baselines-and-regressions) when one exists.

Defaults come from `config('evals.gate')`; class-based evaluations can define `gatePolicy()`; the [CLI](/evals/cli) accepts `--min-score`, `--min-pass-rate`, `--max-regressions`. In Pest a failed gate fails the test; in the CLI it sets exit code `1`.

<Warning>
  An eval where **every** sample errored always fails, regardless of gate configuration — a run that proves nothing must never pass.
</Warning>