Skip to main content
Evaluations attach scores to traces and experiment runs. A score has a name, a value (numeric, categorical, boolean, or text), and a source.

​
Scoring methods

LLM-judge

A model grades an output against instructions or a reference (e.g. faithfulness, helpfulness). Uses claude-opus-4-8 by default.

Assertion

Deterministic checks — exact match, contains, JSON-valid, regex.

Rubric

Multi-criterion scoring with weighted dimensions.

Dataset

Compare outputs against expected values from a dataset.

​
Score sources

SourceWhere it comes from
EVALOffline/online eval runs (the eval worker).
ANNOTATIONHuman review via annotation queues.
APIScores you push yourself.

​
Online vs offline

  • Online scoring runs continuously against live traces (e.g. score every production run for faithfulness).
  • Offline experiments run a prompt/agent over a dataset and score the results, so you can compare versions before shipping.

Experiments

Run an experiment over a dataset and compare it against your baseline.