Scoring Model

​

Assertion scores

• Deterministic assertions score 1.0 (passed) or 0.0 (failed).
• Judge assertions score raw / 10, normalized to 0–1. Pairwise comparisons score win 1.0 / tie 0.5 / loss 0.0.
• Skipped or errored assertions have no score (null) and are excluded from means.
• Every assertion can carry a ->weight(float) (default 1.0).

​

Sample score

• If any gated assertion failed (or errored): the sample scores 0.0 and fails. Gates are binary preconditions — they’re excluded from the score mean, because “the response wasn’t empty” isn’t quality signal.
• Otherwise: score = Σ(weightᵢ × scoreᵢ) / Σ(weightᵢ) over non-gate assertions with scores.
• A sample whose only assertions were passing gates scores 1.0.
• A sample whose agent invocation or evaluation threw is an error: no score, counted separately.

$a->contains($row->expected())          // weight 1.0
  ->toolCalled('lookup_policy')->weight(2.0);   // counts double
// judge at min: 7 scoring 8/10 → 0.8, weight 1.0
// sample score = (1.0×1 + 1.0×2 + 0.8×1) / 4 = 0.95

​

Row result

• pass_rate = passed samples ÷ all samples (errors count in the denominator)
• score_mean, score_stddev = over samples with scores (errors excluded from the mean)

​

Run result

​

The gate

->gate(minScore: 0.8, minPassRate: 0.9, maxRegressions: 0)

• minScore / minPassRate — thresholds on the run aggregates (0–1).
• maxRegressions — enforced against the suite’s baseline when one exists.