ruinator 0.1.0

LLM-augmented mutation-testing framework: inject realistic bugs, see if your tests catch them.

Ruinator

LLM-augmented mutation-testing framework. Inject realistic bugs into your code, see if your tests catch them.

Ruinator is a mutation-testing framework for people who want to know whether their test suite actually catches bugs. It analyzes your source code, introduces realistic bugs (logic errors, off-by-one mistakes, missing guards), and reruns your tests against each mutation. Surviving mutants reveal gaps in your coverage.

Unlike classical rule-based mutation testers (Stryker, PIT, Mutmut), ruinator is plugin-extensible, LLM-augmented, reproducible via seeds + manifests, and CI-native.

Status

Alpha (v0.1). APIs will change. v0.1 ships rule-based mutation with a heuristic equivalence filter; the LLM equivalence judge and creative-mutant slot are scaffolded but not wired into the default pipeline yet.

Quick look

# ruinator.toml
[ruinator]
test_command = "pytest -q"
timeout_seconds = 60
seed = 1
survivor_reruns = 3

[ruinator.scope]
include = ["src/**/*.py"]
exclude = []
# coverage_file = ".coverage"   # optional; when set, only covered lines mutate

[ruinator.selection]
categories = ["logic", "boundary"]
languages = ["python"]
max_mutants = 100
per_file_per_category = 4

[ruinator.llm]
provider = "none"  # "claude" and "litellm" also supported (unused in v0.1 pipeline)

ruinator init                               # scaffold ruinator.toml
ruinator run --config ruinator.toml         # run a mutation campaign
ruinator run --report html --report json    # emit reports alongside the manifest
ruinator run --report markdown --report sarif  # PR-comment + code-scanning outputs
ruinator report .ruinator/runs/latest.json  # re-render reports from a manifest
ruinator replay .ruinator/runs/latest.json  # rerun the same mutants at HEAD

GitHub Action

# .github/workflows/ruinator.yml
name: Mutation testing
on:
  pull_request:

permissions:
  contents: read
  pull-requests: write     # for the PR comment
  security-events: write   # for SARIF upload to Code Scanning

jobs:
  ruinator:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: nauski/ruinator@v0.1.0
        with:
          config: ruinator.toml
          fail-under: '0.80'

The action installs ruinator from PyPI, runs it with your config, emits markdown + SARIF, posts (or updates) a PR comment with the summary, uploads the SARIF to GitHub Code Scanning, and attaches all reports as workflow artifacts. See action.yml for the full input list.

Why ruinator?

	Classical tools (Stryker, PIT)	Mutahunter	LLMorpheus	Ruinator
Cross-language	per-tool	yes	JS only	tree-sitter (many)
LLM-augmented	no	yes	yes	yes (Claude + BYO)
Bug taxonomy	mech operators	none	none	categorized
Reproducible (seed+manifest)	partial	no	no	yes
Plugin framework	per-tool	weak	no	declarative + code
CI baseline-then-diff	no	no	no	yes (v0.2)
Web UI	Stryker only	no	no	v0.2
MCP server	no	no	no	v0.2

How it works

1. Scope. Ruinator resolves your include/exclude globs, parses # ruinator: skip annotations, and (optionally) drops uncovered lines using your coverage.py / lcov report.
2. Candidates. tree-sitter parses each file and queries the AST for candidate nodes per mutator category (conditions, comparisons, boundaries).
3. Sample. A seeded RNG samples N candidates per category per file, up to your budget.
4. Mutate. Built-in mutators (or your plugins) propose a mutation. A creative slot reserved for LLM-proposed bugs is planned for v0.2.
5. Apply. Each mutation lives in its own git worktree under .ruinator/worktrees/<id>/. Your working tree is never touched.
6. Test. Your test_command runs in the worktree with a timeout. Exit code 0 = survived; non-zero = killed.
7. Rerun survivors. Survivors run N more times (default 3) to rule out flaky-pass.
8. Report. JSON manifest (canonical) + HTML report (browser-friendly).

Differentiators

• Plugin framework — declarative YAML mutators (query + prompt template + category) or code mutators (Python class). Discovered via entry-points.
• Bug taxonomy — every mutation carries a category (logic, boundary, and in v0.3 security, concurrency). Budgets allocate per category.
• Reproducibility — mutation IDs are sha256(file, node_range, mutator_id, seed). Replay a manifest and get the same mutant set every time.
• Worktree isolation — one git worktree per mutant. Safe for parallel execution and for security mutants that look like exploit code.
• Cost control — token budget enforced hard. Prompt caching + Haiku/Sonnet model tiering cut token spend aggressively.
• Claude-ready — Anthropic SDK adapter with prompt caching + model tiering, plus LiteLLM for BYO providers. Pipeline defaults to rule-based mutation in v0.1; LLM judging/creation land in v0.2.

Roadmap

• v0.1 (current) — Python + JS, logic + boundary categories, CLI, JSON/HTML reports, heuristic equivalence filter, worktree isolation, manifest cache.
• v0.2 — LLM equivalence judge wired into the pipeline, creative-mutant slot (Claude/LiteLLM proposing realistic bugs), Web UI, MCP server, GitHub Action, SARIF + Markdown reporters, local LLM provider.
• v0.3 — Security + concurrency mutators, multi-file mutations, Go/Rust/TS/Java adapters.
• v0.4 — Auto-test-writer agent (writes tests that kill survivors).
• v1.0 — Stable plugin API (semver), docs site, hosted SaaS beta.

See ROADMAP.md for details.

Contributing

PRs welcome. See CONTRIBUTING.md and CODE_OF_CONDUCT.md.

Development install:

git clone https://github.com/nauski/ruinator
cd ruinator
uv sync --extra dev
uv run pytest
uv run ruinator --help

Prior art & inspiration

• Mutahunter — language-agnostic LLM mutation testing; closest existing tool.
• LLMorpheus — GitHub Next research, JS-only placeholder approach.
• Meta ACH — multi-agent mutation + test generation at industrial scale.
• Stryker, PIT, Mutmut, Cosmic Ray — classical rule-based tools.
• honnibal/claude-skills #mutation-testing — Matthew Honnibal's manual Claude skill; the conceptual starting point.

License

Apache 2.0. See LICENSE.