CI gating - RunAgain

Run your experiments in CI and fail the build when a scorer drops below a threshold — or regresses significantly against a baseline.

The CLI

runagain-eval (in @runagain/cli) triggers an experiment, polls the eval worker for its per-scorer summary, applies your policy, and exits non-zero on failure.

runagain-eval \
  --url "$EVALS_URL" --secret "$EVALS_SHARED_SECRET" \
  --experiment "$EXPERIMENT_ID" --run \
  --min faithfulness=0.9 --min correctness=0.8

Flag	Meaning
`--url`	Eval worker base URL (or `EVALS_URL`).
`--secret`	Shared secret for the worker (or `EVALS_SHARED_SECRET`).
`--experiment`	Experiment id to gate.
`--run`	Trigger the run first, then poll for results.
`--min name=threshold`	Minimum average for a scorer (repeatable).
`--baseline <id>`	Compare against a baseline experiment.
`--fail-on-regression`	Fail on any McNemar-significant regression vs the baseline.
`--poll-seconds`	How long to wait for results (default 60).

--fail-on-regression uses the same significance test as the compare view — so a merge is only blocked when a drop is unlikely to be sampling noise.

GitHub Action

name: eval
on: pull_request
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: pnpm/action-setup@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22, cache: pnpm }
      - run: pnpm install --frozen-lockfile
      - run: >
          pnpm --filter @runagain/cli exec runagain-eval
          --url "$EVALS_URL" --secret "$EVALS_SHARED_SECRET"
          --experiment "$EXPERIMENT_ID" --run
          --baseline "$BASELINE_ID" --fail-on-regression
          --min faithfulness=0.9
        env:
          EVALS_URL: ${{ secrets.EVALS_URL }}
          EVALS_SHARED_SECRET: ${{ secrets.EVALS_SHARED_SECRET }}
          EXPERIMENT_ID: ${{ vars.EXPERIMENT_ID }}
          BASELINE_ID: ${{ vars.BASELINE_ID }}

A non-zero exit fails the job and blocks the merge.