Skip to main content
The people who know whether an answer is good are often not the people who can edit a JSONL file. Cloud lets them propose a change, validate it against a real run, and hand the developer something to commit.
The dataset editor: input and expected per row, with disable and add-a-row controls

​
Where the dataset comes from

It is derived from the samples your runs already reported — no import step, no new payload. The first version anyone edits is the one your eval actually ran. This needs sample detail, so a project running VIZRA_CLOUD_SAMPLES=false has nothing to derive from and editing is unavailable.

​
Editing creates a version, not a change

Editing opens a draft. The current version stays exactly as it was until you publish, so the runs that used it still mean what they said. This is not caution for its own sake. A row’s identity is a hash of its content — input, prior messages and expected value. Change the expected answer and it becomes a different row, because every past score was graded against a different expectation. Keeping the same identity would splice two incompatible gradings into one trend line. So a run of an edited dataset is not compared to a baseline that used the old one. The run page says so plainly rather than showing a delta that means nothing:
Different dataset to the baseline — no overall comparison. This run used dataset v3. 1 row is unchanged and still compared below.
Unchanged rows still line up, and are still compared. That is the only place a comparison means anything.

​
Rows are disabled, not deleted

Deleting is the one operation where a well-meaning edit can quietly destroy coverage, and unlike an edit it leaves nothing behind to show what used to be tested. Disabled rows are excluded from runs, stay visible, and are restored with one click.

​
Getting it back into your repository

Your repository stays canonical. An edit that only ever lives in Cloud is a fork of your test data, so the workflow ends with a file you commit:
  1. Download JSONL — the exact format Dataset::fromJsonl() reads.
  2. Commit it. Your next ordinary run — CI, terminal, anywhere — uses it without knowing any of this happened.
  3. Mark as committed, which records that the version is now in the repo.
Disabled rows are excluded from the export, so committing it removes them from your repository. That is a real deletion arriving by the back door, and the page says so before you download.
The export refuses outright when it cannot write the dataset faithfully — a multi-turn row’s earlier turns never reach Cloud, so the file would be missing part of it. Handing over a plausible file that quietly loses data is worse than refusing; edit those in the repository instead.

​
Why not just let Cloud own the dataset?

Because then Cloud becomes a runtime dependency of your eval suite, and our outage becomes your broken build. Storing variants and handing you a file to merge costs one step and buys you a review — which is also what stops someone deleting half the test set.