Annotations

Annotation queues let humans review traces and attach scores — useful for building gold labels, spot-checking production, or grading edge cases.

​

Create a queue

Annotations → New queue with a name and reviewer instructions.

​

Add items

Add a trace to a queue by its trace_id. Queued items track status (pending → completed).

​

Review

Open a queue item to see the trace, then submit an annotation:

• Name — the score name (e.g. human_rating).
• Value — numeric rating.
• Comment — free-text notes.

Submitted annotations are written as scores with source ANNOTATION, so they show up alongside eval scores on the trace and feed into comparisons.