redherring

Find the CI failures that weren't your fault.

redherring reads the GitHub Actions history your repo already has and finds every run that went red, then turned green when someone re-ran the same commit. Nothing in the code changed between those two attempts, so whatever failed first was not caused by the commit: a flaky test, a network blip, a package registry hiccup, a runner that vanished. redherring opens the failed logs, names the tests (over 20 test-runner output formats) or the infrastructure cause, and ranks them.

Then, when a build fails, redherring why tells you (or your coding agent) which failures are known red herrings and which ones look real.

No server, no signup, nothing to install in CI first. If your repo has Actions history, the first run already has answers.

Real output for a public repo, 28 September 2026. The "AI provider" cause is an AI code-review job failing with "Selected model is at capacity".

How common is this? We ran it on 122 popular repos (1.25 million workflow runs, two weeks). For the median repo about 2% of red runs are red herrings; for a quarter of them it's 7% or more (uv 34%, VS Code 23%, Next.js 15%), and 690 runner-hours went to failed jobs that a re-run fixed. Read the study →

Install

redherring is a Python 3.11+ command-line tool. With uv:

uvx --from git+https://github.com/ixklo/redherring redherring scan OWNER/REPO

or pipx install git+https://github.com/ixklo/redherring.

It needs a GitHub token to read Actions logs. If you use the GitHub CLI, it borrows gh auth token automatically; otherwise set GH_TOKEN. Any token works for public repositories; private ones need actions: read.

redherring scan: what flakes in this repo?

redherring scan                       # the repo of the current checkout
redherring scan vitejs/vite --days 14
redherring scan owner/repo --workflow ci.yml --format md > FLAKY.md

You get:

• Flaky tests: each failed and then passed on a re-run of the same commit, with how often, on how many commits, on which OS (plenty of tests only flake on Windows), and when last seen.
• Flaky jobs with no test named, grouped by cause: network, package registry, rate limited, runner lost, runner environment, job timeout, out of memory, disk full, service startup, test worker crash, GitHub service, AI provider, many tests at once (an environment problem, not ten flaky tests), or unknown.
• What it cost: runner time burned by the failed jobs, and how long commits sat red waiting for someone to press re-run.

--format json gives everything, including example job links, for your own dashboards.

redherring why: is this failure mine?

redherring why https://github.com/owner/repo/actions/runs/123456789
redherring why 123456789 --repo owner/repo --format json

For every failure in the run, one verdict:

Exit codes make it scriptable: 0 all red herrings (or nothing failed), 1 something looks real, 3 unclear, 2 error.

redherring why "$RUN" -q && gh run rerun "$RUN" --failed

For coding agents

Agents lose a lot of time "fixing" failures that were never theirs, and sometimes weaken a perfectly good test to make a flake go green. Add this to your AGENTS.md or CLAUDE.md:

## CI failures
Before changing code because CI failed, run `redherring why <run-url> --format json`.
- `recommendation: "rerun"`: every failure is a known flake, an infrastructure problem, or already
  failing on main. Re-run the failed jobs (`gh run rerun <id> --failed`); don't edit code or tests for it.
- `recommendation: "investigate"`: fix only the findings with `"red_herring": false`.
- Never skip, loosen, or delete a test just because it is listed as flaky.

Or install the ready-made skill: copy skills/redherring/ into your agent's skills folder (for Claude Code, .claude/skills/redherring/). More setups are in examples/.

GitHub Action

Explain every failed CI run, in its job summary and as a comment on the pull request:

# .github/workflows/redherring.yml
name: redherring
on:
  workflow_run:
    workflows: [CI]          # the name of your test workflow
    types: [completed]
permissions:
  actions: read
  contents: read
  pull-requests: write      # only needed for comment: true
jobs:
  why:
    if: github.event.workflow_run.conclusion == 'failure'
    runs-on: ubuntu-latest
    steps:
      - uses: ixklo/redherring@v0.1.0
        with:
          comment: true

The comment is posted once per PR and workflow, then edited in place on later failures, never duplicated.

Or post a weekly flaky-test report:

on:
  schedule: [{ cron: "0 7 * * 1" }]
permissions:
  actions: read
jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: ixklo/redherring@v0.1.0
        with:
          command: scan
          days: "30"

Inputs: command (why or scan), run-id, days (default 14), workflow, ledger, comment, cache (default on: keeps downloaded history between runs, because the Actions token allows about 1,000 API requests an hour), fail-on-real, github-token. Output: recommendation. From the command line, the same comment is redherring why <run> --comment.

Keep your flake history: the ledger

From 1 October 2026 GitHub deletes workflow runs once they pass your log-retention period (90 days by default). Past that, redherring has nothing to read. A ledger keeps the evidence:

redherring scan --ledger .github/flake-ledger.json          # merges new evidence into the file
redherring why "$RUN" --ledger .github/flake-ledger.json    # uses it as extra history

The file is small JSON (one entry per failed job: test names, cause, commit, OS, link). Keep it anywhere durable. You can commit it, or carry it between scheduled runs with actions/cache:

    steps:
      - uses: actions/cache@v6
        with:
          path: flake-ledger.json
          key: redherring-ledger-${{ github.run_id }}
          restore-keys: redherring-ledger-
      - uses: ixklo/redherring@v0.1.0
        with:
          command: scan
          ledger: flake-ledger.json

Entries older than a year are dropped (--keep-days).

How it decides

redherring is deliberately strict about what it calls flaky. Only one kind of evidence counts: the same commit failed and then passed. That is what a re-run is. It does not guess from "failed on a PR, passed on main" or "this test fails a lot", because those are often real failures. A tool that cries wolf is worse than none.

Then, per failed job:

1. Tests. The log is cleaned (timestamps, colour codes) and read by runner-specific parsers. Test ids keep only stable parts (file, suite, name; never line numbers or timings) so the same test lines up across commits and OSes.
2. Infrastructure. If no test is named, the end of the log is matched against specific signatures (ECONNRESET, Could not transfer artifact, The runner has received a shutdown signal, model is at capacity, ...).
3. Not flakiness at all. Roll-up jobs ("all required jobs passed") and PR policy checks are set aside and not counted.

Supported test output:

Missing yours? A parser is one function and a test with a real log excerpt. See CONTRIBUTING.md.

Limits, honestly

• History has an expiry date. Job logs are kept for 90 days by default, and from 1 October 2026 GitHub deletes the workflow runs themselves once they pass the repo's log retention period (90 days by default, and at most 90 days for public repos). --days beyond that finds nothing; use a ledger to keep what was found.
• Only re-runs count. Repos that never press re-run, or retry inside the job without reporting it, show fewer flakes than they have. Playwright, nextest and pytest-rerunfailures retries are recognised when they appear in a failed job's log.
• Big repos cost API requests. A busy monorepo can take a few thousand requests for 30 days. redherring waits out rate limits on its own, and caches everything immutable (attempt job lists, logs) so later runs only fetch what's new.
• Some jobs stay "unknown". In a study of 122 popular repos, nearly half of the red-herring jobs (46%) named neither a test nor a known infrastructure cause. Some logs genuinely say nothing ("exit 1"); others use output formats redherring doesn't parse yet. why treats those as "unclear", never as safe to re-run. Parser contributions fix this one format at a time.
• GitHub Actions only, for now.

Privacy

redherring runs on your machine (or in your own Actions runner) and talks only to the GitHub API. Downloaded logs are cached, compressed, in your user cache folder (%LOCALAPPDATA%\redherring, ~/Library/Caches/redherring, or ~/.cache/redherring; override with REDHERRING_CACHE). Nothing is sent anywhere else.

License

MIT