Skip to main content

ablate

ci license python

ablate measures whether your CLAUDE.md actually does anything — and tells you which parts of it you can delete.

Run a coding agent N times under config A and config B, check outcomes with deterministic assertions, and report the delta with a confidence interval. Then attribute that delta to individual sections. Not "is config A better than config B", but which of these 400 lines can I delete.

No LLM judge, anywhere (ADR-0004). No hosted service, no accounts, no telemetry.

What it looks like

  $ ablate compare CLAUDE.md /dev/null --trials 5
  ablate 0.5.1 · claude-code 2.1.220 · claude-sonnet-5 · 10 tasks × 5 trials × 2 arms

  A  CLAUDE.md                  76 lines ·     748 tokens/turn
  B  (no config)                 0 lines ·       0 tokens/turn

──────────────────────────────────────────────────────────────────────────────────

  PASS RATE                                                         ● INCONCLUSIVE

      A       0.56   █████████████████████░░░░░░░░░░░░░░░░░
      B       0.58   ██████████████████████░░░░░░░░░░░░░░░░

      Δ = -2.0pp   95% CI [-32.0pp, +20.0pp]   BCa, 10k resamples

      Underpowered. The data are consistent with anything from -32.0pp to +20.0pp.
      To resolve to ±5.0pp you would need roughly 5 trials × 316 tasks.

  TOKENS PER TURN                                                     ● REGRESSION

      A     14,122   ██████████████████████████████████████
      B     10,090   ███████████████████████████░░░░░░░░░░░

      Δ = +4,032 (+40.0%)   95% CI [+35.6%, +44.3%]

      A measurably regresses tokens per turn: up 40.0% [+35.6%, +44.3%].

These numbers are synthetic. That output is the bundled demo fixture, not a measurement — reproduce it exactly with ablate demo, which costs nothing. A real headline number goes here once one has been run. Publishing an invented one would be the exact failure this tool exists to catch.

See it for $0

uvx --from ablate-cli ablate demo

No install, no API key, no network, no repo. Prints the full report so you can decide whether to trust the thing before spending anything.

(The distribution is ablate-cli; ablate on PyPI is an unrelated project, so plain uvx ablate will not work. The command you type is still ablate.)

Five-minute path

ablate cost CLAUDE.md                        # $0 — what your config costs per turn
ablate init                                  # $0 — scaffold .ablate/tasks/
# fill in the TODOs in .ablate/tasks/*.yaml
ablate try                                   # ≤ $2 — smallest real run, 18 trials
ablate compare CLAUDE.md /dev/null --trials 5   # ~$28 — the real thing

Then attribute the result to individual sections:

ablate sections CLAUDE.md                    # which sections earn their keep
ablate apply --drop s11,s06 CLAUDE.md --write

ablate sections runs in two stages: it screens every section on a cheap model using a randomized subset design, then confirms a shortlist with dedicated paired A/Bs on the target model. Screening ranks; only the confirmatory stage produces verdicts, and only an EQUIVALENT verdict licenses a deletion. ablate apply archives whatever it removes under .ablate/removed/, so no deletion is irreversible.

compare, try and sections need the claude CLI on PATH and ANTHROPIC_API_KEY set. Trials run claude --bare so that the config under test is the only config in play, and --bare reads only that variable and the Bedrock/Vertex/Foundry equivalents — it never reads the login from claude /login. Being logged in interactively is not enough; without a key every trial fails "Not logged in", and ablate now refuses to start rather than discovering that eighteen trials later.

If you would rather use an existing subscription login, --injection native drops --bare. That works, but your own hooks, plugins, MCP servers and user-level CLAUDE.md then enter every trial, so the delta reflects your whole environment rather than the file under test. It is a weaker claim; say so if you report the number.

These commands call the agent once per trial inside an isolated git worktree, spending real money against your own account. Each prints the run count, the estimated cost, and the effect size the design can actually resolve before executing anything, and stops at --max-spend (default $25).

Why not just eyeball it?

Because three 2026 studies measured exactly this, and disagreed with intuition.

  • Gloaguen et al. (ETH Zürich), arXiv 2602.11988 — four coding agents over SWE-bench Lite and a 138-repo benchmark. LLM-generated context files reduced task success by ~3% and raised inference cost by >20%. Human-written ones gained ~4%.
  • arXiv 2605.10039 — a factorial study of four config structure variables across 1,650 Claude Code sessions. No detectable effect on instruction adherence for any variable, after multiple-testing correction.
  • arXiv 2601.20404 — AGENTS.md across 10 repos and 124 PRs: 28.6% median runtime reduction and 16.6% fewer output tokens, with comparable completion.

The effects are small, sometimes negative, and mostly show up in cost rather than in success. That is not something you can see by reading your config.

How it decides

Every comparison resolves to exactly one of four verdicts, from the confidence interval and an equivalence margin m (default ±5pp; --margin to change).

Verdict When Means
🔴 REGRESSION interval sits below −m A measurably worse than B
🟢 IMPROVEMENT interval sits above +m A measurably better than B
🔵 EQUIVALENT interval fits inside ±m Bounded. Safe to prefer the cheaper config.
🟡 INCONCLUSIVE anything else Not enough data to say — here's what it would cost

EQUIVALENT and INCONCLUSIVE are opposite findings, and conflating them is the specific thing that makes people distrust eval tools. "We looked hard and there is nothing there" is what licenses a deletion. "We didn't look hard enough" licenses nothing. They never share a colour, a sentence, or a code path — and every INCONCLUSIVE verdict ends with the tasks, trials and runs that would resolve it.

What it costs

Command Cost Time What
ablate demo $0 3 s Bundled run. No API key, no network.
ablate cost $0 instant Per-section token cost. Exact arithmetic.
ablate try ≤ $2 ~6 min 3 tasks × 3 trials on Haiku. Smoke test, not a measurement.
ablate compare ~$28 ~25 min 8 tasks × 5 trials × 2 arms on Sonnet.
ablate sections ~$110 ~90 min Screen on Haiku + confirm on Sonnet, K=12.

Every spending command prints its estimate and its resolvable effect size before it starts, and honours --max-spend. In CI the ceiling is mandatory and breaching it fails the job.

What it can't tell you

The honest limits, up front rather than after you have spent $28.

  • Pass rate is a guardrail, not a headline. Published pass-rate effects of config changes are 3–4 percentage points. A default 8-task × 5-trial run resolves about 32 points. It will usually return INCONCLUSIVE on pass rate, and says so instead of pretending otherwise (ADR-0010).
  • Token cost is where the signal is. The same run resolves a ~16% change in tokens per turn, and measured cost effects are 16–28%. That asymmetry is why tokens are the primary metric.
  • Below 8 tasks there is no confidence interval at all. ablate init scaffolds three and ablate try runs three — enough to prove the plumbing, not to measure anything.
  • Δ is an average over your suite, for your agent at your model version. It is not a general claim about the config.
  • Screening is a ranking, not a measurement. Ridge shrinks coefficients toward zero (~75% of true magnitude in simulation) and screening intervals cover ~88% rather than the nominal 95%. Read the order of the screening table, not its numbers — which is why the confirmatory stage is not optional.
  • Attribution is section-level with line ranges, not per-line. --granularity bullet makes it literally per-item and roughly triples the cost.

Method

Every interval comes from one engine: a paired cluster bootstrap with BCa correction, pairing at task level, resampling tasks and then trials within arm (ADR-0005). Coverage is validated empirically, not assumed — an A/A calibration confirms ~95% of intervals contain zero when the truth is zero.

The shortcut this avoids is pooling every trial into a binomial CI, which treats correlated trials as independent. For a single arm's pass rate that interval comes out ~2× too narrow. For the paired difference the error is smaller — 1.05× at moderate between-task heterogeneity, 1.3× when tasks respond very differently — because pairing already cancels task difficulty. Both figures are measured, not asserted.

What a task looks like

A task is a prompt plus deterministic checks, pinned to a commit. ablate init scaffolds three; this repo's own suite is in .ablate/tasks/.

id: convention-cost-json
description: Add a --format json option to `ablate cost`.

setup: uv venv -q && uv pip install -q -e ".[dev]"
prompt: |
  `ablate cost` prints a human-readable table. Add a `--format text|json`
  option. Follow the conventions already used by the other commands.

assert:
  - id: tests-pass
    type: exit_code
    run: .venv/bin/python -m pytest tests/test_cli.py -q

  - id: test-added
    type: files_touched
    must_include: ["tests/**"]

  - id: no-bare-except
    type: absent
    pattern: 'except\s*:'
    in: diff

tags: [convention, smoke]

Six assertion types: exit_code, files_touched, present/absent, lint, script. All deterministic, all runnable offline, all checkable by hand.

In CI

- uses: ablate-dev/ablate@v1
  with:
    config: CLAUDE.md
    baseline: origin/${{ github.base_ref }}
    max-spend: 15          # required — no default, on purpose
    fail-on: never         # regression | inconclusive | never
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Posts a sticky PR comment whose footer states the MDE for the run — how large a regression would have had to be before that green check could have caught it. Start with fail-on: never; the first thing most teams learn is that their suite is underpowered for pass rate, which is worth knowing before it gates a merge. Full example in docs/examples/ablate.yml.

Prior art

Generously and specifically: this stands on other people's work.

  • promptfoo — the assertion vocabulary and YAML shape are modelled on it.
  • inspect-ai (UK AISI) — --epochs, bootstrap scoring, and the solver/scorer split.
  • probe-and-refine-tuning (arXiv 2606.20512) — the screening-then-confirm method this project's ablation stage is built on.
  • SWE-bench harness — the task-instance schema.
  • AgentLinter — static CLAUDE.md linting; complementary, and the zero-signup npx framing is worth copying.

Full comparison in docs/01-prior-art.md.

Install

uvx --from ablate-cli ablate demo

The distribution is ablate-cli because ablate on PyPI belongs to an unrelated deep-learning reporting tool. The command you type is still ablate.

From source:

uv venv && uv pip install -e ".[dev]" && source .venv/bin/activate
ablate --help

Status

Milestone 4 of 4 — the planned v1 surface is complete, and 0.5.0 is on PyPI. Two caveats still apply to any number this tool prints:

  • ablate demo replays a synthetic fixture, labelled on every run. It shows you what a report looks like; it measures nothing.
  • Q2 is unresolved. Trials deliver the config as an appended system prompt, which is not byte-identical to how Claude Code loads a project CLAUDE.md. Until tools/q2_injection_equivalence.py runs, read every result as measuring a system-prompt appendix rather than a CLAUDE.md — see Q2.

Full design rationale in docs/; start with docs/00-verdict.md. Progress in CHANGELOG.md.

Repository layout

Path What
ablate/ The CLI — tasks, assertions, runner, adapters, statistics, reports.
.ablate/tasks/ This repo's own task suite.
docs/ Design docs and ADRs. Read before proposing anything structural.
tests/ pytest suite — runs against real git repos, not mocks.

Contributing

See CONTRIBUTING.md. Tests and lint must pass:

uv run pytest -q
uv run ruff check .

License

Apache-2.0

Metadata

Release files for ablate-cli 0.5.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ablate-cli 0.5.1
File Size Uploaded Provenance Actions
ablate_cli-0.5.1.tar.gz 286.2 kB Without provenance Details for ablate_cli-0.5.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for ablate-cli 0.5.1
File Interpreter ABI Platform Actions
ablate_cli-0.5.1-py3-none-any.whl Python 3 none any Details for ablate_cli-0.5.1-py3-none-any.whl

Total release size: 407.8 kB

Release files / ablate_cli-0.5.1.tar.gz

Download URL ablate_cli-0.5.1.tar.gz
Size 286.2 kB
Tags Source
SHA-256 checksum
How to use checksums
c9447e1c73a66785b70151423ff8ef4532d9f6f3674409530c977f88f6ade5f9
BLAKE2b-256 checksum
How to use checksums
fa78887b904fdc8634599ef47f5ad487ec6b1adfe3617ce2272532a24b4a6625
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.11.9 {"installer":{"name":"uv","version":"0.11.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / ablate_cli-0.5.1-py3-none-any.whl

Download URL ablate_cli-0.5.1-py3-none-any.whl
Size 121.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
791bfde07f466fa3d1962cab3e459b40ce1fb27a62629a93f863fcba3ef34906
BLAKE2b-256 checksum
How to use checksums
e0363b14208051342e5949a618c5ed5293778e99c2e765233024a8febc20a989
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.11.9 {"installer":{"name":"uv","version":"0.11.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for ablate-cli 0.5.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.5.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page