Datasets

A dataset yields rows; each row is one thing your agent should handle well. Pass anything below to ->dataset(...) in Pest, or return it from a class-based evaluation’s dataset() method.

JSONL — the preferred format

One JSON object per line, streamed lazily (large files never load fully into memory):

evals/support.jsonl

{"input": "What is your refund policy?", "expected": "30 days"}
{"input": "Do you ship to France?", "expected": "France", "category": "shipping"}
{"messages": [{"role": "user", "content": "Hi, I ordered a lamp"}, {"role": "assistant", "content": "How can I help?"}, {"role": "user", "content": "Can I return it?"}], "expected": "30 days"}

Recognised keys:

Key	Meaning
`input`	The user prompt (single-turn rows)
`messages`	`[{role, content}, …]` — the final entry must be a `user` turn and becomes the prompt; earlier turns are replayed as conversation context (multi-turn)
`expected`	Free-form reference data — a string, or an object read with `$row->expected('key')`
`hash`	Optional. Overrides the computed content hash with one you supply — for datasets that already know their row identities. Must be a sha256 hex digest or the row is rejected
anything else	Lands in `$row->meta('key')`

hash is identity, not data — a column of that name is not available as meta, and a value that isn’t a 64-character sha256 digest aborts the read. Rename the column if you meant it as your own field.

Malformed lines fail loudly with their line number — no silent row-dropping.

CSV

evals/support.csv

prompt,expected,category
"What is your refund policy?","30 days",billing
"Do you ship to France?",,shipping

Header row required; prompt and expected columns by default (both configurable via Dataset::fromCsv($path, inputColumn: ..., expectedColumn: ...)); other columns become meta. CSV exists so non-developers can own datasets in a spreadsheet.

Arrays and Eloquent

Inline rows

->dataset([
    'A bare string is a prompt',
    ['input' => 'What is your refund policy?', 'expected' => '30 days'],
])

From your database

use Vizra\Evals\Dataset\Dataset;

->dataset(Dataset::fromEloquent(
    Ticket::query()->where('resolved', true),
    fn ($ticket) => ['input' => $ticket->question, 'expected' => $ticket->resolution],
))

Production conversations

The differentiated one — build the dataset from conversations your agent actually had, stored by the Laravel AI SDK’s conversation feature:

Replay real traffic

expect(SupportBot::class)->toPassEval(fn ($eval) => $eval
    ->fromConversations(take: 50)
    ->judge('At least as helpful as the reply we actually sent.')
);

Each stored conversation becomes a multi-turn row: the latest user turn is the prompt, prior turns replay as context, and the reply your agent actually gave is exposed as $row->expected() — ideal for judging new prompts against real production answers. The standalone Dataset::fromConversations(SupportBot::class) form adds ->latest(), ->take(n), and ->where(...) refinement.

Row identity

Every row gets a content hash over input + messages + expected (not meta). It’s how baseline comparison joins rows across runs — stable across file reordering, unaffected by meta changes, changed only when the row’s actual content changes.

Getting Started

Core Concepts

Writing Evals

Running & CI

Dashboard

Migration

JSONL — the preferred format

CSV

Arrays and Eloquent

Production conversations

Row identity

​JSONL — the preferred format

​CSV

​Arrays and Eloquent

​Production conversations

​Row identity

JSONL — the preferred format

CSV

Arrays and Eloquent

Production conversations

Row identity