ActionsBench

A labeled benchmark and evaluation harness for GitHub Actions workflow security scanners.

Status: initial release (0.1.1, pre-1.0); 0.2.0 in preparation. The corpus is a small hand-built set (44 cases). Version 0.2.0 carries all 44 labels as ai-reviewed following a parallel agent audit, plus a third scanner adapter (poutine). No label has independent human review. Published 0.1.1 still contains the original 44 proposed labels. Nothing here is a result yet. It is public early so the labels can be challenged, not because it is finished.

Why this exists

Several open-source scanners analyze GitHub Actions workflows for security weaknesses, and they disagree widely. In a 2026 study of 9 scanners on 2,722 real workflows, one tool reported 107 injection findings where another reported 1,778 on the same files. The same study found no verified, labeled dataset of workflows with known weaknesses, so precision and recall could not be computed (arXiv 2601.14455).

ActionsBench provides the missing piece:

1. A labeled corpus of minimized workflows, each tagged with the weakness it contains, or marked clean.
2. A harness that scores any scanner against the corpus, per weakness class.
3. Versioned, reproducible releases so results stay comparable over time.

It is not a scanner, and it does not rank tools. It reports per-class evidence.

Full motivation, scope and risks: docs/PRD.md.

Install

pip install actionsbench      # or: uv tool install actionsbench
actionsbench validate         # works from any directory: the corpus is bundled

The package bundles the corpus, so it can validate and score without a clone. Archived releases have a DOI on Zenodo (this version; the concept DOI 10.5281/zenodo.23175472 always resolves to the latest). The code is MIT and the corpus is CC BY 4.0.

Quickstart (from a clone)

Requires Python 3.11+ and uv.

uv sync
uv run actionsbench validate      # check every case in corpus/cases
uv run actionsbench stats         # cases per weakness class
uv run actionsbench run --tool zizmor      # needs uvx; add --config persona=auditor for another setting
sh scripts/install-actionlint.sh           # pinned, checksum-verified download into .tools/
uv run actionsbench run --tool actionlint
sh scripts/install-poutine.sh              # pinned, checksum-verified download into .tools/
uv run actionsbench run --tool poutine
uv run actionsbench score results/zizmor-1.30.1-regular/findings.json

run executes a scanner that has a reviewed adapter (zizmor, scored on six weakness classes; actionlint, four; poutine, three) and writes a findings file; raw output goes to results/ (gitignored). score takes any normalized findings file (schema: findings.schema.json), so a scanner without an adapter can still be scored by producing that file. More adapters are planned (see plan.md).

Poutine runs local analysis with an explicit empty config and its version check disabled. Its job- and execution-step anchors expose limits of region matching; see the mapping review.

Run the commands from the repository root, or pass --corpus explicitly. The poutine adapter and ai-reviewed status described here land in 0.2.0; PyPI 0.1.1 does not include them.

How scoring works

• A tool is judged only on the weakness classes it declares in scope. Anything outside that scope is reported as a coverage gap, never as a miss or a false alarm.
• Cases the tool did not run on are excluded, not counted as clean.
• Matching is one-to-one (class, file, and the same step, job or top-level key; use --match line for exact lines), so duplicate findings do not inflate recall.
• Results are per class. There is no overall score on purpose.

Details: docs/methodology.md.

Repository layout

A note on the corpus

The cases contain deliberately vulnerable workflows. They are inert: GitHub only runs workflows under the repository root's .github/workflows/, and these live inside corpus/cases/. Payloads are benign placeholders. Never copy them into a real repository. See SECURITY.md.

Contributing

Labels are claims, and they are meant to be challenged. See CONTRIBUTING.md for adding cases, challenging a label, and the development setup.

All pull requests target the dev branch. main and the v* tags are the released state (see ADR 0004).

License

• Code (src/, tests/, tooling): MIT.
• Corpus (corpus/, the labeled cases and their labels): CC BY 4.0. Reuse is welcome with attribution; please cite ActionsBench and the corpus version you used.