> ## Documentation Index
> Fetch the complete documentation index at: https://docs.runagain.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# CI gating

> Block a merge when an experiment regresses, with the runagain-eval CLI.

Run your [experiments](/app/experiments) in CI and fail the build when a scorer drops
below a threshold — or regresses significantly against a baseline.

## The CLI

`runagain-eval` (in `@runagain/cli`) triggers an experiment, polls the eval worker for
its per-scorer summary, applies your policy, and exits non-zero on failure.

```bash theme={null}
runagain-eval \
  --url "$EVALS_URL" --secret "$EVALS_SHARED_SECRET" \
  --experiment "$EXPERIMENT_ID" --run \
  --min faithfulness=0.9 --min correctness=0.8
```

| Flag                   | Meaning                                                         |
| ---------------------- | --------------------------------------------------------------- |
| `--url`                | Eval worker base URL (or `EVALS_URL`).                          |
| `--secret`             | Shared secret for the worker (or `EVALS_SHARED_SECRET`).        |
| `--experiment`         | Experiment id to gate.                                          |
| `--run`                | Trigger the run first, then poll for results.                   |
| `--min name=threshold` | Minimum average for a scorer (repeatable).                      |
| `--baseline <id>`      | Compare against a baseline experiment.                          |
| `--fail-on-regression` | Fail on any **McNemar-significant** regression vs the baseline. |
| `--poll-seconds`       | How long to wait for results (default 60).                      |

<Tip>
  `--fail-on-regression` uses the same [significance test](/app/experiments#compare) as the
  compare view — so a merge is only blocked when a drop is unlikely to be sampling noise.
</Tip>

## GitHub Action

```yaml theme={null}
name: eval
on: pull_request
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: pnpm/action-setup@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22, cache: pnpm }
      - run: pnpm install --frozen-lockfile
      - run: >
          pnpm --filter @runagain/cli exec runagain-eval
          --url "$EVALS_URL" --secret "$EVALS_SHARED_SECRET"
          --experiment "$EXPERIMENT_ID" --run
          --baseline "$BASELINE_ID" --fail-on-regression
          --min faithfulness=0.9
        env:
          EVALS_URL: ${{ secrets.EVALS_URL }}
          EVALS_SHARED_SECRET: ${{ secrets.EVALS_SHARED_SECRET }}
          EXPERIMENT_ID: ${{ vars.EXPERIMENT_ID }}
          BASELINE_ID: ${{ vars.BASELINE_ID }}
```

A non-zero exit fails the job and blocks the merge.