Skip to main content
RunAgain can post to Slack when something goes wrong in a project — a failed eval, a tool error, or a trace error. Alerts are event-driven: they fire as the data lands, not on a schedule.

​
Setup

Alerting has two layers, matching how the rest of RunAgain is scoped:
  1. Connect Slack once per workspace. In Workspace settings click Add to Slack and approve. This authorizes the bot for the org, shared by every project.
  2. Pick or create the default channel. Still in Workspace settings, choose an existing channel or create one — this is where alerts go unless a project overrides it.
  3. Configure the rule per project. In Project settings → Integrations → Slack alerts, enable alerts, choose what to report, and optionally point that project at its own channel.

​
Destination channel

Alerts go to a Slack channel you pick — or one RunAgain creates for you. Both the workspace default and a per-project override offer the same two choices: Select existing. Pick any channel the RunAgain bot can see. Public channels appear automatically; a private channel appears once the bot is in it. Create new. Name the channel, choose public or private, and pick who to add. RunAgain creates it, invites the people you ticked, and points alerts at it in one step — the bot joins by creating the channel, so there’s no separate invite step for it.
Whoever sets this up is added to the new channel automatically, matched from their RunAgain account email to their Slack account. Their row is ticked and can’t be unticked — you’d otherwise create an alert channel you can’t see.
People you can add are the members of the connected Slack workspace, with anyone who is also a member of your RunAgain workspace flagged as a teammate so they’re easy to find. Deactivated accounts and bots are filtered out. A project’s own choice always wins; leave it unset and the project follows the workspace default.
Connected Slack before channel creation was added? Reconnect Slack in Workspace settings to grant the new permissions. Alerts keep posting in the meantime — you just can’t create channels until you do.

​
What you can report

The eval-failure threshold lives on the rule (default 0.5). A boolean score fails at 0; a numeric or categorical score fails when its value is below the threshold.

​
Digests, not floods

Events for a project are coalesced into one message per short window. A broken deploy that emits 5,000 tool errors becomes a single “Tool errors (5,000)” message with the first few examples and an “and N more” link — never 5,000 pings. Each line deep-links back to the trace in the app.

​
How it works

Alerting is a push pipeline, so there’s no polling delay:
  • The ingest worker flags errored spans as they’re written and pushes them onto an alert queue.
  • The evals worker emits a candidate whenever a scorer writes a failing-eligible score.
  • A queue consumer matches each candidate against the project’s rule (the only place the rule is read), then hands survivors to a per-project aggregator that debounces a short window and posts one digest.

​
Configuration

The web app needs SLACK_CLIENT_ID / SLACK_CLIENT_SECRET for the workspace connection. The Slack app must request the incoming-webhook, chat:write, channels:read and groups:read bot scopes to list channels and post, plus channels:manage, groups:write, users:read and users:read.email to create channels and add people. The evals worker uses APP_BASE_URL to build the deep links in each alert, and the ingest and evals workers share a runagain-alerts queue.