Experiments

An experiment runs your task over a dataset, records each output, and scores it — so you can compare versions and catch regressions.

​

Create an experiment

In Experiments → New experiment, choose a dataset and set:

• Prompt template — e.g. Answer concisely. {{input}}
• Task model — e.g. claude-opus-4-8
• Trials per item — run each item N times to measure non-determinism (default 1).
• Scorers — pick from the built-in catalog (evaluations): heuristic, LLM-judge, and embedding scorers, each with its own arguments.

​

Run it

Trigger the run from the experiment page. The eval worker executes the task for each dataset item, writes a run item (input, output, latency, tokens, cost, and any error), and applies the scorers.

​

Compare

The experiment view shows per-scorer averages and counts. For a rigorous diff, open Experiments → Compare and pick a baseline and a candidate:

• Pass rate + Wilson bands — each scorer’s pass rate (value ≥ 0.5) with a 95% Wilson confidence interval, so small samples don’t read as over-confident.
• McNemar significance — a paired test over the items both runs share, so a change is only called an improvement or regression when it’s unlikely to be sampling noise (p < 0.05).
• Not-comparable — items whose data changed between the two runs (by content hash) are excluded, separating a real regression from an edited dataset.
• Changed items — the individual pass/fail flips, with input and both outputs side by side.

​

Score means & effect size (continuous)

Pass rate collapses a graded score to a coin flip. The second table treats scores as continuous values in 0–1 and reports:

• Bootstrap 95% CI — the mean of each run with a percentile confidence interval from 2,000 resamples (seeded, so the interval is reproducible).
• Paired t-test & Wilcoxon — two-sided significance on the per-item differences. The t-test assumes roughly normal differences; the Wilcoxon signed-rank test is the rank-based fallback when they aren’t. * marks p < 0.05, ** p < 0.01.
• Cohen’s d — the standardized effect size (0.2 small · 0.5 medium · 0.8 large), so you can tell a statistically significant change from a practically meaningful one.

This is the per-metric comparison table you’d bring to a provider-selection or ship/no-ship decision — CI, p-value, and effect size side by side.

​

Failure rate

Items that errored — timeouts, empty responses, task failures — are reported as a failure rate per run and excluded from the score averages, so a provider that fails often can’t look good by only being scored on the requests it happened to answer. Compare the failure rates alongside the quality scores.