Evaluations

Evaluations attach scores to traces and experiment runs. A score has a name, a value (numeric, categorical, boolean, or text), and a source.

​

Scoring methods

​

Heuristic scorers

Deterministic, model-free scorers compare an output to a reference (or check the output alone). Each returns a value in 0–1. No API key required.

​

Built-in LLM judges

Ready-made judges you can attach by name — no prompt to write. Each returns a score in 0–1 where higher is better (for safety judges, higher = safer). Grading uses claude-sonnet-5 by default; override per scorer. The battle judge compares the output against the reference (expected) as two candidates, in a random A/B order each call to cancel positional bias, and maps the verdict back: 1 = output wins, 0.5 = tie, 0 = reference wins.

​

RAG evaluation

For retrieval-augmented answers, pass the retrieved passages as the dataset item’s metadata.context. These judges then grade grounding and retrieval quality: Use faithfulness for a quick grounding score and faithfulness_claims when you want the hallucination rate — which specific claims aren’t supported. Retrieval ranking (IR metrics) is scored deterministically (no model call). Put the ranked retrieved ids (or source URLs) in the output; supply the ground truth one of two ways:

• Binary — a flat set of relevant ids in the item’s expected value (JSON array or comma-separated). Every listed id counts as relevant.
• Graded (qrels) — the item’s relevance labels: [{ "ref": "…", "grade": 0–3 }]. When present these take precedence over expected; grade 0 is an explicit negative, higher grades mean more relevant. See Datasets → Relevance labels.

All accept an optional k cutoff (default: the number of retrieved ids). Automated search metrics (no LLM) run on the query/response/sources triple with no model call — a fast, free first-pass filter: Entity extraction is a lightweight heuristic (not full NER) — a proxy for “did the response mention what the query is about”, best read alongside the LLM judges.

​

Embedding similarity

The embedding_similarity scorer measures semantic closeness between the output and the expected value (a.k.a. answer similarity) — useful when wording differs but meaning should match. Cosine similarity is rescaled into 0–1. Two BERTScore proxies reuse the same embedding to score search responses without a ground-truth answer: (Classic BERTScore uses token-level embeddings + greedy matching, which needs a local model; these fill the same role via the project’s embedding provider.) Choose an embedding provider per project on the Evals page → Embeddings: Each provider needs its API key on the eval worker (OPENAI_API_KEY, COHERE_API_KEY, GEMINI_API_KEY via wrangler secret). Embedding scorers are skipped when no provider is configured. Each score records its embedding tokens + USD cost like the LLM judge.

​

Score sources

​

Score configs

A score config pins the value schema for a named score so a value can never land out of range. Define them on the Evals page → Score configs:

• NUMERIC — optional min/max bounds.
• CATEGORICAL — an allowed set of category values.
• BOOLEAN — 0 or 1.
• TEXT — free-form, up to 500 characters.

Human annotations (and, over time, evals) are validated against the matching config before the score is written — an out-of-range or unknown-category value is rejected. This is also the primitive behind CI threshold gating.

​

Project LLM judge

Add an LLM-as-judge to a project in one step — no scorer to author. On the Evals page, add an LLM judge eval and choose:

• What to judge:
  • Output — grades the run’s final answer against the original request. Writes a score named llm_judge.output.
  • Tool calls — grades whether the agent used its tools correctly and efficiently (right tools, sane arguments, no thrashing, errors handled). Writes llm_judge.tool_calls.
  • Tool selection — grades whether the agent chose the right tools for the task, given the tools it had available (correct tool picked, no wrong or unnecessary tools, none missed) — judged apart from execution quality. The judge is shown the union of the run’s available tools (from the request’s tool definitions) alongside the calls it made. Writes llm_judge.tool_selection.
• Scope: all runs, a specific agent, or an attribute match — see scoping.
• Coverage: all runs, or a sample (a percentage of runs) to control cost.
• Judge model: optional; defaults to claude-sonnet-5.

The eval worker scores recent completed runs in the background (never in the request path) and dedupes so each run is judged once. Scores are source = EVAL, valued 0–1, with the judge’s one-line reason as the comment. Results surface as a judge badge on the trace (colored by score) and in the trace’s Scores table. Requires ANTHROPIC_API_KEY on the eval worker. Every judge score records the judge model, token count, USD cost, latency, and the raw model reply of the grading call (priced from the model tables in @runagain/core) — the judge’s own execution is observable, so you can see and budget what evaluation itself costs. The trace’s Scores table shows a model · tokens · cost · latency summary per eval score.

​

Tool-call trajectory eval

Score how well an agent used its tools — deterministically, no model call. Add a Tool-call trajectory eval on the Evals page. For each sampled run it reconstructs the ordered tool results and writes a trajectory.tool_success score (0–1) = the fraction of tool calls that succeeded, surfaced as a badge on the trace.

• Recovery invalidation (on by default) — an errored tool call is not counted as a failure if a later call to the same tool succeeds. So an agent that hits an error, retries, and ultimately gets there scores as if it succeeded. Turn it off to count every error strictly.
• Scope & coverage — grade all runs, a specific agent, or an attribute match, on all matching runs or a sampled %. Model-free, so it runs regardless of the judge key.

The score’s metadata records status (ok / recovered / failed), total calls, errors, and recovered/unrecovered counts. For subjective trajectory quality (“were these the right tools, in the right order?”), pair it with the LLM judge tool_calls target. Tool-error recovery rate. The same eval also emits a trajectory.recovery_rate score — of the tool calls that errored, the fraction the agent recovered from (a later call to the same tool succeeded). It’s the recovery signal on its own, independent of the overall success rate: a run can have a low success rate but a high recovery rate (it hit errors but always worked through them). Runs with no tool errors are skipped (nothing to recover), so the score reflects only runs that actually hit an error.

​

Approval-guard eval

Check that the agent obtained approval before invoking a guarded (sensitive) tool — deterministic, no model call. Add an Approval guard eval on the Evals page and configure:

• Guarded tools — the tools that must not run without prior approval (e.g. delete_file, send_email, charge_card).
• Approval tools — the tool calls that count as an approval / confirmation step (e.g. request_approval, confirm). In a human-in-the-loop agent this is whatever call the agent makes to ask before acting.

For each sampled run it reconstructs the ordered tool-call sequence and requires each guarded call to be preceded by an approval call. Approvals are consumed one-to-one — a single “request approval” green-lights the next guarded action, not every action after it, so N guarded calls need N prior approvals. It writes a guardrail.approval score (0–1) = the fraction of guarded calls that were approved; runs that invoked no guarded tool are skipped. Metadata records status (ok / violated), guarded-call count, approved count, and violations — and a violation trips eval alerts like any failing score.

​

Tool-sequence eval

Check that a run called the expected tools — deterministic, no model call. Add a Tool sequence eval on the Evals page, list the expected tool names, and pick a mode:

• Called in order — the expected tools must appear in this relative order in the run’s tool-call sequence (other calls may come in between). Turn on Strict contiguous order to require them back-to-back with nothing between. Good for flows like search → book or plan → apply.
• Called at all — each expected tool must be called at least Min calls per tool times (default 1), order-independent. Good for “the agent must always call confirm”.

For each sampled run it reconstructs the ordered tool-call sequence and scores the fraction of expected steps satisfied: tool_sequence.order or tool_sequence.contains (0–1). Partial matches earn partial credit (e.g. two of three expected calls, in order → 0.67). A run that made none of the expected calls scores 0 — absence is exactly what this eval catches, so (unlike the trajectory / approval evals) it isn’t skipped. Metadata records status (ok / partial / failed), the mode, the expected list, how many matched, and which are missing — and a low score trips eval alerts like any failing score. For subjective ordering quality (was this the smart order?), use a project LLM judge on tool_calls instead. Scoping a broadly-applied tool-sequence eval to the right agent matters: a run with no tool calls scores 0, so pointing it at chat-only traffic will fail every run.

​

Structured-output eval

When an agent’s final answer is structured JSON, evaluate it by its properties — and get alerted when its shape drifts. Add a Structured output eval on the Evals page:

• Property checks (metadata matching) — assert on fields of the JSON by path (result.status, items[0].id). Operators: exists, is not empty, equals, contains, matches regex, is of type (string / number / boolean / array / object / null). Writes a structured_match score = the fraction of checks that passed, with the failing path op → actual list as the comment.
• Shape-drift alert — compares each run’s JSON shape (every field path + type, with array indices collapsed so length changes don’t count) against the same agent’s previous run. If a field is added, removed, or changes type, it writes a structured_output.changed score.
• Scope & coverage — all runs, a specific agent, or an attribute match, on all matching runs or a sampled %.

The diff is surfaced on the trace: a panel lists the added (+), removed (−), and retyped (~) fields versus the previous run, and structured_match shows as a header badge. Runs in the background on the eval worker; requires no model call for property checks (drift alerting is also model-free).

​

Pass thresholds

Any eval can carry an optional pass threshold (passThreshold + passDirection) that turns its numeric score into an explicit pass/fail:

• gte (default) — passes when value ≥ threshold. Use for quality scores (judge, trajectory, tool-sequence, structured) where higher is better.
• lte — passes when value ≤ threshold. Use for run baselines, whose score value is the raw metric (runtime ms, cost, span count), so ≤ 30000 means “fail runs slower than 30s”.
• BOOLEAN scores pass at 1, fail at 0, regardless of direction.

A failing score renders red wherever it’s shown and fires the eval-failure alert. The per-eval threshold overrides the project-wide alert threshold for that eval; evals with no threshold keep the default band coloring and the project threshold. When several evals write the same score name, the score passes only if it clears every such eval’s threshold (strictest-wins). Baseline values are never compared against the project-wide 0–1 threshold — only against their own per-eval limit.

​

Run status

Every run gets a status derived from its spans — the headline “did this run finish cleanly?” signal, computed model-free (shared between the trace UI and the run.health eval): The status is primarily whether the run returned an output at the end. Chat agents are treated differently: because a conversational run’s “output” is a reply to the latest user turn, a chat run with no reply is reported as no reply rather than the task-agent “died mid-run” phrasing. A run is detected as chat when its transcript carries more than one user turn. When run baselines are enabled, any non-OK status is also persisted as a run.health score so it’s queryable and alertable.

​

Run score

The run score (0–1) is a single headline per run that folds eval quality with run health. It starts from the mean of the applied eval dimensions (faithfulness / trajectory / schema) — or a nominal 1 when nothing scored the run, so health alone can move it — then health gates it:

• errored run → 0 (regardless of eval quality);
• incomplete run (no output / truncated / ended-on-tool-call / no-reply) → ×0.4;
• a clean, healthy run keeps its eval mean (or 1 when unevaluated).

Tool-call quality already flows in through the trajectory.tool_success eval dimension when that eval is configured, so the run score has no separate tool-error factor. It surfaces as the headline number in the traces-list evals column and on the trace detail, with the issues that moved it on hover. This means a run with a problem scores low even when no eval was configured, and a run that errored can’t be rescued by good eval dimensions.

​

Online vs offline

• Online scoring runs continuously against live traces (e.g. score every production run for faithfulness, or the project LLM judge above).
• Offline experiments run a prompt/agent over a dataset and score the results, so you can compare versions before shipping.