Skip to main content
RunAgain can post to Slack when something goes wrong in a project — a failed eval, a tool error, or a trace error. Alerts are event-driven: they fire as the data lands, not on a schedule.

​
Setup

Alerting has two layers, matching how the rest of RunAgain is scoped:
  1. Connect Slack once per workspace. In Workspace settings click Add to Slack and approve. This authorizes the bot for the org, shared by every project.
  2. Configure the rule per project. In Project settings → Integrations → Slack alerts, enable alerts, choose what to report, and pick the destination channel.

​
Destination channel

Once Slack is connected, each project’s rule has a Destination channel selector. Leave it on Workspace default to post to the channel you chose when connecting, or pick any other channel — RunAgain posts the digest there with the workspace bot token.
To post to a private channel, invite the bot to it first — type /invite @RunAgain in that channel. Private channels only appear in the selector after the bot has been added.
Connected Slack before this was added? Reconnect Slack in Workspace settings so the bot gets the channel permissions — until then, alerts fall back to the default channel.

​
What you can report

The eval-failure threshold lives on the rule (default 0.5). A boolean score fails at 0; a numeric or categorical score fails when its value is below the threshold.

​
Digests, not floods

Events for a project are coalesced into one message per short window. A broken deploy that emits 5,000 tool errors becomes a single “Tool errors (5,000)” message with the first few examples and an “and N more” link — never 5,000 pings. Each line deep-links back to the trace in the app.

​
How it works

Alerting is a push pipeline, so there’s no polling delay:
  • The ingest worker flags errored spans as they’re written and pushes them onto an alert queue.
  • The evals worker emits a candidate whenever a scorer writes a failing-eligible score.
  • A queue consumer matches each candidate against the project’s rule (the only place the rule is read), then hands survivors to a per-project aggregator that debounces a short window and posts one digest.

​
Configuration

The web app needs SLACK_CLIENT_ID / SLACK_CLIENT_SECRET for the workspace connection. The Slack app must request the incoming-webhook, chat:write, channels:read, and groups:read bot scopes (the last three power the channel picker and posting). The evals worker uses APP_BASE_URL to build the deep links in each alert, and the ingest and evals workers share a runagain-alerts queue.