> ## Documentation Index
> Fetch the complete documentation index at: https://docs.runagain.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Evals

> The dedicated page for managing a project's online evals — add evals, scope them, and see what's applied.

The **Evals** page is where you manage every online evaluation for a project. It
replaces the old *Settings → Evals* tab: instead of a handful of per-project
toggles, you keep a **list of applied evals**, add as many as you like, and
**scope** each one to the runs it should grade.

Open it from the sidebar (**Evals**), under a project.

## Applied evals

The top of the page lists the evals configured for this project. Each row shows
its **type**, **scope**, **coverage**, recent **average score**, and a
**live / paused** pill — click the pill to toggle it, or the ✕ to delete. Filter
the list by typing in **Search**, or with the **type** and **status** chips; the
header shows how many are live out of the total.

The average score is the mean of the eval's score over the last 30 days, colored
green (≥ 0.90), amber (≥ 0.80), or red below that. Run baselines have no single
0–1 score, so they show **—**.

### Auto Evals

At the top of the page, **Auto Evals** asks **Claude Sonnet** to read your **most
recent run** and suggest online evals to add — each mapped to a real eval kind and
skipping evals you already have. Review the list and **Add** the ones you want in one
click; from a [trace](/app/traces) the same panel analyzes *that* run instead.

### Add an eval

**Add eval** opens a three-step wizard:

1. **Type** — pick how the eval scores a run (the four types below).
2. **Configure & scope** — set the type's options, choose the **scope** (all runs,
   an agent, or an attribute), and drag **coverage** to sample a percentage of
   matching runs.
3. **Review** — name it and create it. Scoring begins on the next matching run;
   choose **Add another** to queue up more without leaving the page.

### Pass threshold

Any eval can carry an optional **pass threshold** with a direction: a run **passes**
when its score is **≥** the threshold (for quality scores like a judge or trajectory) or
**≤** the threshold (for **run baselines**, whose value is a raw metric — ms, dollars,
or span count). A score on the failing side of the threshold turns **red** everywhere it's
shown (the evals list and the trace) and fires the [eval-failure alert](/app/alerts). Leave
it empty to keep the default band coloring and the project-wide alert threshold.

This is how you make an eval **hard-fail** a run — e.g. a **baseline** runtime eval with
`≤ 30000` fails runs slower than 30s, or a judge with `≥ 0.8` fails low-quality answers.

### Eval kinds

| Kind                     | Writes                                                 | What it does                                                                                                                                                                                                                                                             |
| ------------------------ | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **LLM judge**            | `llm_judge.<target>`                                   | A model grades the run's **output**, its **tool calls**, or its **tool selection** (were the right tools chosen for the task, given what was available). Pick a judge model (defaults to `claude-sonnet-5`). See [Evaluations](/concepts/evaluations#project-llm-judge). |
| **Tool-call trajectory** | `trajectory.tool_success` + `trajectory.recovery_rate` | Model-free score of how well the agent used its tools, with optional [recovery invalidation](/concepts/evaluations#tool-call-trajectory-eval). Also emits a **tool-error recovery rate** — of the tool calls that errored, the fraction the agent recovered from.        |
| **Approval guard**       | `guardrail.approval`                                   | Model-free check that the agent obtained **approval before invoking guarded (sensitive) tools**. Configure the guarded tools and the tools that count as an approval step. See [Evaluations](/concepts/evaluations#approval-guard-eval).                                 |
| **Tool sequence**        | `tool_sequence.order` / `tool_sequence.contains`       | Model-free check that expected tools were called — in a required **relative order**, or simply **called at all** (≥ N times). See [Evaluations](/concepts/evaluations#tool-sequence-eval).                                                                               |
| **Structured output**    | `structured_match` / `structured_output.changed`       | Property checks on JSON answers plus [shape-drift alerting](/concepts/evaluations#structured-output-eval).                                                                                                                                                               |
| **Run baseline**         | `baseline.*`                                           | Flags abnormal runtime, cost, or span count per agent — the [run baselines](/concepts/baselines) online eval.                                                                                                                                                            |

## Scope — where an eval runs

Every eval has a **scope** that decides which runs it grades:

* **All runs** — every run in the project.
* **Agent** — only runs from a specific agent (matched on the run's `agent_name`).
  The picker lists the agents seen on your recent traces.
* **Attribute** — only runs whose **agent metadata** has a given `key = value`.
  Traces carry arbitrary metadata; pick a key from the ones seen recently and set
  the value to match on.

This lets you, for example, run an expensive LLM judge only on your
`checkout-agent`, run a cheap trajectory score on everything, and run a strict
structured-output check only where `metadata.tier = premium`.

<Note>
  Attribute scope matches against the metadata merged across a run's spans. A run
  matches when any of its spans carries the `key = value` pair.
</Note>

## Coverage

Each eval also has a **coverage** (sample rate) — grade **all** matching runs or a
**percentage** of them to control cost. Coverage applies on top of scope: a 20%
coverage on an agent-scoped eval grades one in five of *that agent's* runs.

## Migrating from Settings

Projects that had the old per-project eval toggles enabled were migrated
automatically: each enabled toggle became an **All runs** eval on this page with
its existing config preserved. Nothing to do — they keep scoring exactly as before,
and you can now re-scope, disable, or add more.

## Embeddings

Choose the **embedding provider** for the `embedding_similarity` scorer (OpenAI,
Cohere, or Gemini). Set the matching API key on the eval worker; leave it disabled
to skip embedding scorers. See [Embedding similarity](/concepts/evaluations#embedding-similarity).

## Score configs

Pin the **value schema** for a named score — numeric range, categories, boolean, or
text — so a score value can never land out of range. Human annotations and evals are
validated against the matching config. See [Score configs](/concepts/evaluations#score-configs).

## Seeing what ran

The **trace** and **session** detail pages show an **evals** chip row: every eval
that's in scope for that run, marked ✓ when it produced a score and `· pending`
when it's in scope but hasn't scored yet. This is the quick "was evaluation running
on this run?" signal.