Skip to main content
A dataset is a versioned collection of items — each an input and an optional expected output — that you run experiments against.

​
Building a dataset

  • Create a dataset in Datasets → New dataset.
  • Add items manually (input as JSON or text, plus expected), or capture them from real traces you want to turn into regression cases.
  • Datasets are versioned, so an experiment records exactly which version it ran against.

​
Relevance labels (qrels)

For retrieval / search evaluation, an item can carry graded relevance judgments — which documents or source URLs are relevant to the query, and how relevant. Add them in the item form as JSON:
  • ref — a document id or source URL your system retrieves.
  • grade — 0 (irrelevant, an explicit negative) up to 3 (perfectly relevant).
The deterministic IR ranking metrics (ndcg, recall_at_k, average_precision, …) use these grades when present, and fall back to treating the expected value as a flat set of relevant ids otherwise. Graded labels unlock graded nDCG (better rankings of more-relevant documents score higher) and correct recall.

​
Stratify by query type

Search quality varies by the kind of query, so a representative dataset should cover a mix. Tag each item with a query type so you can document your coverage and read scores per stratum: The dataset page shows the composition — the count and share of each query type — so you can weight your set toward what your production traffic actually looks like.

​
Using a dataset

Run an experiment over a dataset: RunAgain executes your prompt/agent for each item, records the output as a run item, and scores it with your chosen method(s). Compare the result against your baseline to catch regressions before shipping.

Experiments

Score a dataset run and diff it against the baseline experiment.