ruleprof 0.1.1

Ablate your CLAUDE.md / AGENTS.md rules to see which ones actually change agent behavior vs. just cost tokens.

ruleprof

Your CLAUDE.md says "use snake_case." Delete that line and nothing changes — the model already does it. Your CLAUDE.md also says "start every file with # RP-MARK-7731." Delete that and adherence drops to zero. One of those rules earns its tokens. The other is pure cost you pay on every single call. ruleprof tells you which is which.

It does this the only way you actually can: it removes a rule, re-runs the agent, and checks whether the agent still follows it.

| verdict | rule                    | tokens / $·run | adherence with → without | note                                      |
|---------|-------------------------|---------------:|--------------------------|-------------------------------------------|
| CUT     | Engineering philosophy  |   165 / $0.0002 | 100% → 100%             | agent obeys this with or without the rule |
| CUT     | Naming                  |    24 / $0.0000 | 100% → 100%             | agent obeys this with or without the rule |
| KEEP    | File header convention  |    40 / $0.0000 | 100% → 0%               | removing it drops adherence — earns its tokens |

189 tokens = $0.0002/run the agent obeys anyway. At 1,000 runs/day that's $69/yr
for zero behavior change.

That 189 is paid on every request against this config, and it's small only because the example config is small. On a real 4KB CLAUDE.md the dead weight is usually most of the file — and the dollar line scales with it. Pass --input-price to price a model ruleprof doesn't know, or let it infer from --model (Haiku $1, Sonnet $3, Opus $5 per Mtok).

Why this exists

Static analyzers (ccmd.dev, AgentLint) price each line and flag vague wording. What they can't tell you is whether a rule does anything — that needs a run. The research backs the suspicion that much of it doesn't: Gloaguen et al. ("Evaluating AGENTS.md", arXiv 2602.11988, 2026) found context files don't generally improve task success while adding 20–23% to inference cost on average, with LLM-generated ones causing outright performance drops in 5 of 8 settings tested. ruleprof turns "probably bloated" into a per-rule KEEP/CUT you can act on.

The rule it applies: a rule earns its tokens only if removing it moves the agent off the behavior it prescribes. If the agent does the thing anyway, the rule is inert.

Install

pip install ruleprof        # needs the `claude` CLI on PATH and Python 3.11+

Use

ruleprof init               # scaffold ruleprof.toml from your CLAUDE.md
# fill in one check per rule, then:
ruleprof run --md report.md

A spec maps each rule to a check — a shell command, run in the agent's output directory, that exits 0 when the rule was followed:

task = "Create solution.py with a function that sums a list of numbers. Then stop."
model = "haiku"
trials = 3

[[check]]
block = "File header convention"        # substring matched against the rule's heading
cmd = "test \"$(grep -m1 . solution.py)\" = '# RP-MARK-7731'"   # exit 0 = followed

[[check]]
block = "Naming"
cmd = "grep -qE '^\\s*def [a-z][a-z0-9_]*\\(' solution.py"

ruleprof splits your config into blocks at each ## heading, then for each block with a check runs the agent trials times with the full config and trials times with that block removed. The adherence gap is the verdict. --json report.json writes the full data.

What it can and can't tell you

• Bind a check, get a verdict. Blocks with no check are listed with their token cost but marked unmeasured — ruleprof won't invent a signal it can't observe.
• Verdicts are for your model and your task. A rule that's dead weight for Haiku on a scaffolding task may matter for a weaker model or a trickier one. Profile the setup you actually run.
• Fuzzy rules give fuzzy answers. "Write clean code" has no behavioral signature to check, so it lands in the unmeasured pile. That's the honest result, not a gap.
• Verdicts carry a 90% confidence interval. A rule is only called KEEP or CUT when the whole interval for its adherence delta lands on one side of the line. Run too few trials and the interval is wide, so you get INCONCLUSIVE with a suggested trial count, not a coin-flip verdict. A clean flip resolves in ~4 trials; a genuinely neutral rule needs ~6 to confirm it's dead weight. Tune with --confidence.
• It costs real money. Each profile is (1 + measured_blocks) × trials × 2 agent runs. Trials buy resolution: 2 is enough to gauge cost on Haiku (the example ran $0.27 there) but usually too few to clear the interval — bump to 6 for verdicts.

Reproducing the example

examples/ holds a CLAUDE.md with three rules of known character — a non-default marker, a redundant style rule, and a block of generic values prose — plus the spec that profiles them. At 6 trials it returns KEEP for the marker and CUT for the other two, with the full 90% intervals printed; the committed examples/report.md is that run ($0.81). Drop to --trials 2 and the same rules come back INCONCLUSIVE — the signal is there but the interval is too wide to call, which is the point.

ruleprof run --spec examples/ruleprof.toml --trials 6

License

MIT.