Skip to main content
Run baselines are online evals that score every agent run against the recent history of the same agent. Instead of comparing two runs by hand, RunAgain learns what “normal” looks like for each agent and flags the runs that stand out. Three metrics are scored per run:
  • Runtime — wall-clock duration of the trace.
  • Cost — total cost of the run.
  • Span count — number of spans (a proxy for tool-loop / retry behavior).

​
How it works

  • Runs are grouped by agent_name. Each new run is compared against that agent’s most recent runs (the baseline window).
  • A run is only scored once the agent has enough history — the learning phase. Until then the trace shows baseline learning (n/5) instead of a verdict.
  • Each metric gets an ok / warning / drift verdict, shown as a badge on the trace detail (vs baseline: Runtime … Cost … Spans …).
  • Results are persisted as trace-level eval scores (baseline.duration_ms, baseline.total_cost, baseline.span_count), so they show up in the Scores table and can be queried like any other score.

​
Baseline methods

All five express the tolerance k in the same units (standard-deviation equivalents), so one threshold behaves consistently:
  • Robust z-score (median / MAD) — default. Median ± k·MAD. Outlier-resistant, best for skewed, spiky latency/cost.
  • Z-score — mean ± k·σ. Classic, but sensitive to the outliers you’re hunting.
  • IQR / Tukey fences — nonparametric, based on the interquartile range.
  • Percentile rank — flags runs beyond the normal-equivalent tail of the baseline.
  • EWMA control chart — trend-aware; the center tracks a moving average.

​
Configure

1

Open project Settings

Go to Settings → Run baselines.
2

Enable and tune

Turn baselines on, pick a method, and set the learning-phase minimum, the baseline window (how many prior runs to compare against), and the threshold k.
“Same agent” is matched by agent_name. Set it on your spans so runs group correctly (the Claude Code integration sets claude-code automatically).
A perfectly constant metric produces no verdict (zero spread → no false drift), and runs below the learning-phase minimum are never scored.

​
Run health

The same eval pass also checks each run’s health and records a run.health score when a run is incomplete — it made model calls but produced no final response. The status names the likely cause: error, truncated (hit the token limit), dangling_tool_call (ended asking for a tool), or no_output. Healthy runs get no score, so the presence of a run.health score means the run had a problem. This surfaces on the traces list (a no output badge) and the trace detail (a banner).