Applied evals
The top of the page lists the evals configured for this project. Each row shows its type, scope, coverage, recent average score, and a live / paused pill — click the pill to toggle it, or the ✕ to delete. Filter the list by typing in Search, or with the type and status chips; the header shows how many are live out of the total. The average score is the mean of the eval’s score over the last 30 days, colored green (≥ 0.90), amber (≥ 0.80), or red below that. Run baselines have no single 0–1 score, so they show —.Auto Evals
At the top of the page, Auto Evals asks Claude Sonnet to read your most recent run and suggest online evals to add — each mapped to a real eval kind and skipping evals you already have. Review the list and Add the ones you want in one click; from a trace the same panel analyzes that run instead.Add an eval
Add eval opens a three-step wizard:- Type — pick how the eval scores a run (the four types below).
- Configure & scope — set the type’s options, choose the scope (all runs, an agent, or an attribute), and drag coverage to sample a percentage of matching runs.
- Review — name it and create it. Scoring begins on the next matching run; choose Add another to queue up more without leaving the page.
Pass threshold
Any eval can carry an optional pass threshold with a direction: a run passes when its score is ≥ the threshold (for quality scores like a judge or trajectory) or ≤ the threshold (for run baselines, whose value is a raw metric — ms, dollars, or span count). A score on the failing side of the threshold turns red everywhere it’s shown (the evals list and the trace) and fires the eval-failure alert. Leave it empty to keep the default band coloring and the project-wide alert threshold. This is how you make an eval hard-fail a run — e.g. a baseline runtime eval with≤ 30000 fails runs slower than 30s, or a judge with ≥ 0.8 fails low-quality answers.
Eval kinds
Scope — where an eval runs
Every eval has a scope that decides which runs it grades:- All runs — every run in the project.
- Agent — only runs from a specific agent (matched on the run’s
agent_name). The picker lists the agents seen on your recent traces. - Attribute — only runs whose agent metadata has a given
key = value. Traces carry arbitrary metadata; pick a key from the ones seen recently and set the value to match on.
checkout-agent, run a cheap trajectory score on everything, and run a strict
structured-output check only where metadata.tier = premium.
Attribute scope matches against the metadata merged across a run’s spans. A run
matches when any of its spans carries the
key = value pair.Coverage
Each eval also has a coverage (sample rate) — grade all matching runs or a percentage of them to control cost. Coverage applies on top of scope: a 20% coverage on an agent-scoped eval grades one in five of that agent’s runs.Migrating from Settings
Projects that had the old per-project eval toggles enabled were migrated automatically: each enabled toggle became an All runs eval on this page with its existing config preserved. Nothing to do — they keep scoring exactly as before, and you can now re-scope, disable, or add more.Embeddings
Choose the embedding provider for theembedding_similarity scorer (OpenAI,
Cohere, or Gemini). Set the matching API key on the eval worker; leave it disabled
to skip embedding scorers. See Embedding similarity.
Score configs
Pin the value schema for a named score — numeric range, categories, boolean, or text — so a score value can never land out of range. Human annotations and evals are validated against the matching config. See Score configs.Seeing what ran
The trace and session detail pages show an evals chip row: every eval that’s in scope for that run, marked ✓ when it produced a score and· pending
when it’s in scope but hasn’t scored yet. This is the quick “was evaluation running
on this run?” signal.