ablate-cli 0.5.0

Measure whether your CLAUDE.md actually does anything.

ablate

ablate measures whether your CLAUDE.md actually does anything — and tells you which parts of it you can delete.

Run a coding agent N times under config A and config B, check outcomes with deterministic assertions, and report the delta with a confidence interval. Then attribute that delta to individual sections. Not "is config A better than config B", but which of these 400 lines can I delete.

No LLM judge, anywhere (ADR-0004). No hosted service, no accounts, no telemetry.

What it looks like

  $ ablate compare CLAUDE.md /dev/null --trials 5
  ablate 0.4.0 · claude-code 2.1.220 · claude-sonnet-5 · 10 tasks × 5 trials × 2 arms

  A  CLAUDE.md                  76 lines ·     748 tokens/turn
  B  (no config)                 0 lines ·       0 tokens/turn

──────────────────────────────────────────────────────────────────────────────────

  PASS RATE                                                         ● INCONCLUSIVE

      A       0.56   █████████████████████░░░░░░░░░░░░░░░░░
      B       0.58   ██████████████████████░░░░░░░░░░░░░░░░

      Δ = -2.0pp   95% CI [-32.0pp, +20.0pp]   BCa, 10k resamples

      Underpowered. The data are consistent with anything from -32.0pp to +20.0pp.
      To resolve to ±5.0pp you would need roughly 5 trials × 316 tasks.

  TOKENS PER TURN                                                     ● REGRESSION

      A     14,122   ██████████████████████████████████████
      B     10,090   ███████████████████████████░░░░░░░░░░░

      Δ = +4,032 (+40.0%)   95% CI [+35.6%, +44.3%]

      A measurably regresses tokens per turn: up 40.0% [+35.6%, +44.3%].

These numbers are synthetic. That output is the bundled demo fixture, not a measurement — reproduce it exactly with ablate demo, which costs nothing. A real headline number goes here once one has been run. Publishing an invented one would be the exact failure this tool exists to catch.

See it for $0

ablate demo

No API key, no network, no repo. Prints the full report so you can decide whether to trust the thing before spending anything.

Five-minute path

ablate cost CLAUDE.md                        # $0 — what your config costs per turn
ablate init                                  # $0 — scaffold .ablate/tasks/
# fill in the TODOs in .ablate/tasks/*.yaml
ablate try                                   # ≤ $2 — smallest real run, 18 trials
ablate compare CLAUDE.md /dev/null --trials 5   # ~$28 — the real thing

Then attribute the result to individual sections:

ablate sections CLAUDE.md                    # which sections earn their keep
ablate apply --drop s11,s06 CLAUDE.md --write

ablate sections runs in two stages: it screens every section on a cheap model using a randomized subset design, then confirms a shortlist with dedicated paired A/Bs on the target model. Screening ranks; only the confirmatory stage produces verdicts, and only an EQUIVALENT verdict licenses a deletion. ablate apply archives whatever it removes under .ablate/removed/, so no deletion is irreversible.

compare, try and sections need the claude CLI on PATH and an authenticated session or ANTHROPIC_API_KEY. They call it once per trial inside an isolated git worktree, spending real money against your own account. Each prints the run count, the estimated cost, and the effect size the design can actually resolve before executing anything, and stops at --max-spend (default $25).

Why not just eyeball it?

Because three 2026 studies measured exactly this, and disagreed with intuition.

• Gloaguen et al. (ETH Zürich), arXiv 2602.11988 — four coding agents over SWE-bench Lite and a 138-repo benchmark. LLM-generated context files reduced task success by ~3% and raised inference cost by >20%. Human-written ones gained ~4%.
• arXiv 2605.10039 — a factorial study of four config structure variables across 1,650 Claude Code sessions. No detectable effect on instruction adherence for any variable, after multiple-testing correction.
• arXiv 2601.20404 — AGENTS.md across 10 repos and 124 PRs: 28.6% median runtime reduction and 16.6% fewer output tokens, with comparable completion.

The effects are small, sometimes negative, and mostly show up in cost rather than in success. That is not something you can see by reading your config.

How it decides

Every comparison resolves to exactly one of four verdicts, from the confidence interval and an equivalence margin m (default ±5pp; --margin to change).

Verdict	When	Means
🔴 REGRESSION	interval sits below −m	A measurably worse than B
🟢 IMPROVEMENT	interval sits above +m	A measurably better than B
🔵 EQUIVALENT	interval fits inside ±m	Bounded. Safe to prefer the cheaper config.
🟡 INCONCLUSIVE	anything else	Not enough data to say — here's what it would cost

EQUIVALENT and INCONCLUSIVE are opposite findings, and conflating them is the specific thing that makes people distrust eval tools. "We looked hard and there is nothing there" is what licenses a deletion. "We didn't look hard enough" licenses nothing. They never share a colour, a sentence, or a code path — and every INCONCLUSIVE verdict ends with the tasks, trials and runs that would resolve it.

What it costs

Command	Cost	Time	What
ablate demo	$0	3 s	Bundled run. No API key, no network.
ablate cost	$0	instant	Per-section token cost. Exact arithmetic.
ablate try	≤ $2	~6 min	3 tasks × 3 trials on Haiku. Smoke test, not a measurement.
ablate compare	~$28	~25 min	8 tasks × 5 trials × 2 arms on Sonnet.
ablate sections	~$110	~90 min	Screen on Haiku + confirm on Sonnet, K=12.

Every spending command prints its estimate and its resolvable effect size before it starts, and honours --max-spend. In CI the ceiling is mandatory and breaching it fails the job.

What it can't tell you

The honest limits, up front rather than after you have spent $28.

• Pass rate is a guardrail, not a headline. Published pass-rate effects of config changes are 3–4 percentage points. A default 8-task × 5-trial run resolves about 32 points. It will usually return INCONCLUSIVE on pass rate, and says so instead of pretending otherwise (ADR-0010).
• Token cost is where the signal is. The same run resolves a ~16% change in tokens per turn, and measured cost effects are 16–28%. That asymmetry is why tokens are the primary metric.
• Below 8 tasks there is no confidence interval at all. ablate init scaffolds three and ablate try runs three — enough to prove the plumbing, not to measure anything.
• Δ is an average over your suite, for your agent at your model version. It is not a general claim about the config.
• Screening is a ranking, not a measurement. Ridge shrinks coefficients toward zero (~75% of true magnitude in simulation) and screening intervals cover ~88% rather than the nominal 95%. Read the order of the screening table, not its numbers — which is why the confirmatory stage is not optional.
• Attribution is section-level with line ranges, not per-line. --granularity bullet makes it literally per-item and roughly triples the cost.

Method

Every interval comes from one engine: a paired cluster bootstrap with BCa correction, pairing at task level, resampling tasks and then trials within arm (ADR-0005). Coverage is validated empirically, not assumed — an A/A calibration confirms ~95% of intervals contain zero when the truth is zero.

The shortcut this avoids is pooling every trial into a binomial CI, which treats correlated trials as independent. For a single arm's pass rate that interval comes out ~2× too narrow. For the paired difference the error is smaller — 1.05× at moderate between-task heterogeneity, 1.3× when tasks respond very differently — because pairing already cancels task difficulty. Both figures are measured, not asserted.

What a task looks like

A task is a prompt plus deterministic checks, pinned to a commit. ablate init scaffolds three; this repo's own suite is in .ablate/tasks/.

id: convention-cost-json
description: Add a --format json option to `ablate cost`.

setup: uv venv -q && uv pip install -q -e ".[dev]"
prompt: |
  `ablate cost` prints a human-readable table. Add a `--format text|json`
  option. Follow the conventions already used by the other commands.

assert:
  - id: tests-pass
    type: exit_code
    run: .venv/bin/python -m pytest tests/test_cli.py -q

  - id: test-added
    type: files_touched
    must_include: ["tests/**"]

  - id: no-bare-except
    type: absent
    pattern: 'except\s*:'
    in: diff

tags: [convention, smoke]

Six assertion types: exit_code, files_touched, present/absent, lint, script. All deterministic, all runnable offline, all checkable by hand.

In CI

- uses: ablate-dev/ablate@v1
  with:
    config: CLAUDE.md
    baseline: origin/${{ github.base_ref }}
    max-spend: 15          # required — no default, on purpose
    fail-on: never         # regression | inconclusive | never
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Posts a sticky PR comment whose footer states the MDE for the run — how large a regression would have had to be before that green check could have caught it. Start with fail-on: never; the first thing most teams learn is that their suite is underpowered for pass rate, which is worth knowing before it gates a merge. Full example in docs/examples/ablate.yml.

Prior art

Generously and specifically: this stands on other people's work.

• promptfoo — the assertion vocabulary and YAML shape are modelled on it.
• inspect-ai (UK AISI) — --epochs, bootstrap scoring, and the solver/scorer split.
• probe-and-refine-tuning (arXiv 2606.20512) — the screening-then-confirm method this project's ablation stage is built on.
• SWE-bench harness — the task-instance schema.
• AgentLinter — static CLAUDE.md linting; complementary, and the zero-signup npx framing is worth copying.

Full comparison in docs/01-prior-art.md.

Install

uvx --from ablate-cli ablate demo

The distribution is ablate-cli because ablate on PyPI belongs to an unrelated deep-learning reporting tool. The command you type is still ablate.

From source:

uv venv && uv pip install -e ".[dev]" && source .venv/bin/activate
ablate --help

Status

Milestone 4 of 4 — the planned v1 surface is complete. Three things are done in this repo but not yet true in the world:

• Not on PyPI yet, so uvx doesn't work. The publish job is written and gated on a repository variable.
• ablate demo replays a synthetic fixture, labelled on every run.
• Nothing has been run against a live agent yet. Until tools/q2_injection_equivalence.py runs, treat every result as measuring a system-prompt appendix rather than a CLAUDE.md — see Q2.

Full design rationale in docs/; start with docs/00-verdict.md. Progress in CHANGELOG.md.

Repository layout

Path	What
ablate/	The CLI — tasks, assertions, runner, adapters, statistics, reports.
.ablate/tasks/	This repo's own task suite.
docs/	Design docs and ADRs. Read before proposing anything structural.
tests/	pytest suite — runs against real git repos, not mocks.

Contributing

See CONTRIBUTING.md. Tests and lint must pass:

uv run pytest -q
uv run ruff check .

License

Apache-2.0