Skip to main content
Annotation queues let humans review traces and attach scores — useful for building gold labels, spot-checking production, or grading edge cases.

​
Create a queue

Annotations → New queue with a name and reviewer instructions.

​
Add items

Add a trace to a queue by its trace_id. Queued items track status (pending → completed).

​
Review

Open a queue item to see the trace, then submit an annotation:
  • Name — the score name (e.g. human_rating).
  • Value — numeric rating.
  • Comment — free-text notes.
Submitted annotations are written as scores with source ANNOTATION, so they show up alongside eval scores on the trace and feed into comparisons.

​
Judge alignment

The Annotations page shows a Judge alignment table: for each score name, how well the LLM judge agrees with human labels. To calibrate a judge, annotate traces using the same score name the judge writes — the two are paired per trace and scored:
  • Agreement — how often judge and human land on the same pass/fail (≥ 0.5).
  • Correlation — Pearson correlation of the raw values (graded agreement).
  • Bias — mean(judge − human); positive means the judge is more lenient.
Human values are averaged across annotators (consensus) before pairing. Use this to decide whether a judge is trustworthy, or which prompt needs tightening.