ruleprobe

Find out which of your agent rules actually fire.

You have written rules for your coding agent. A CLAUDE.md, an AGENTS.md, a house style your team argued over. You do not know whether any of them changed what the agent did. ruleprobe reads the transcripts your agent already wrote and counts the observable things that happened in them.

Sixty seconds

uvx ruleprobe report

That reads ~/.claude/projects/**/*.jsonl (and ~/.codex/sessions/**/*.jsonl, if Codex wrote any), runs the detectors over them, and prints:

detector                                 hits  sessions    of   share  note
---------------------------------------------------------------------------
cache-hygiene/compact                       0         0   339       0%  unobserved
cache-hygiene/model-switch                 11        11   339       3%
secrets/secret-in-write                     3         2   339       1%
transcript-hygiene/unfiltered-find          0         0   339       0%  unobserved
transcript-hygiene/whole-file-cat          89        22   339       6%
verification/no-verify                      0         0   339       0%  unobserved

That is a real run, over one week of one person's transcripts. Two detectors have never fired on this machine, which is itself the finding: unfiltered-find and no-verify are guarding against something that is not happening, and the rules behind them are paying rent in the context window for nothing.

share is the fraction of sessions the detector fired in at least once. unobserved means it has never fired in the window. promote?, above a 30 percent share, means it is common enough that either the rule is worth stating more loudly or the rule is wrong. Both notes stay blank until there are twenty measured sessions, because a share over five sessions is noise.

Nothing is sent anywhere, no model is asked anything, nothing is written to disk, and the same transcript gives the same answer every time. Python 3.9 or newer, standard library only.

Other groupings, and a window:

uvx ruleprobe report --by repo --since 30      # last 30 days, one line per repository
uvx ruleprobe report --by stance --stance commits=conventional   # grouped by configuration
uvx ruleprobe report --root ./transcripts      # a directory of your own
uvx ruleprobe report --rules ./docs/rules      # bind detectors to rule files, and name the gaps
uvx ruleprobe detectors                        # what would run
uvx ruleprobe corpus                           # how good each detector is, over the labelled corpus
uvx ruleprobe report --json                    # the same numbers as data, rows included

A transcript does not record the configuration it ran under, so --stance dimension=variant is how you say what it was. It is repeatable, it is what --by stance groups on, and it is what a detector's gate: block reads: a gated detector with no stance passed never fires, and ruleprobe detectors names the stance each one is waiting for.

Sixty seconds on a rule of your own

The six above are generic. Your rules are not, and measuring one takes no Python: write a detector beside them as data. This is a real run over the example transcript and the example rules in this repository, so it is reproducible from a clone:

cat docs/rules/house-style.md

---
rule: house-style
detector:
  id: house-style/sudo-install
  event: tool_use
  when:
    command: {starts_with: [sudo, pip]}
---

# House style

Install with `uv`, never with `sudo pip`.

ruleprobe report --root docs --rules docs/rules

detector                                 hits  sessions    of   share  note
---------------------------------------------------------------------------
cache-hygiene/compact                       0         0     1       0%
cache-hygiene/model-switch                  0         0     1       0%
house-style/sudo-install                    2         1     1     100%
secrets/secret-in-write                     0         0     1       0%
transcript-hygiene/unfiltered-find          0         0     1       0%
transcript-hygiene/whole-file-cat           0         0     1       0%
verification/no-test-run                    1         1     1     100%
verification/no-verify                      0         0     1       0%

rules: 2 measured, 1 dark, 1 unmeasured
  measured   house-style                 docs/rules/house-style.md
  dark       secrets                     docs/rules/secrets.md: a credential that never reaches a file leaves no shape in a transcript
  measured   verification                docs/rules/verification.md
  unmeasured working-style               docs/rules/working-style.md

Three lines of that report are the point. house-style/sudo-install is a rule of the reader's own, firing. secrets is dark by choice: its front matter carries opt_out: <reason>, because the rule is about a credential that never reaches a file and a transcript only shows what did. working-style is unmeasured: it has neither a detector nor an opt-out, and saying so is the only way an author sees the gap. Nothing fails; a report is evidence, not a gate.

Point --rules at whatever directory your own rules live in, and drop the same entries into .ruleprobe/detectors.yaml at the root of a repository, or into ~/.config/ruleprobe/detectors.yaml for the ones you want everywhere. Both are found without a flag; --no-config skips them.

Writing a detector

An entry is id, rule, event, when, and an optional gate. event is one of tool_use, assistant_text and session, and it says what a hit is counted against. when is a matcher: a mapping in which every key must hold, composed with any, all and not.

A tool use. The shape is a command, a tool name, or an argument:

- id: house-style/wide-grep
  rule: house-style
  event: tool_use
  when:
    command:
      name: grep
      none_of: [-n, --include, --exclude]
      arg_count: {max: 1}

An assistant message. One hit per message the pattern matches:

- id: voice/hedged-verdict
  rule: voice
  event: assistant_text
  when:
    message:
      role: assistant
      final: true
      regex: "(?i)(should (now )?work|I think it works)"

A whole session. absent counts a session in which something never happened, which is the only way to measure a rule that asks for something to be done; order counts one event followed by another; change counts a field that differs from the event before it:

- id: verification/no-test-run
  rule: verification
  event: session
  when:
    absent:
      of:
        command: {name: [pytest, tox, nox]}
      scope: session

The matchers, in one list: tool (name, glob), arg (field, regex, path_glob, contains, equals, exists), command (name, starts_with, contains, none_of, arg_count, sole_segment, redirect, unparsed, regex), git (subcommand, args_any, args_none, token_prefix), env (name, command), text (source, regex, contains), message (role, final, regex, contains), kind, and the three session matchers order, absent and change. ruleprobe/matchers.py documents each in one line. The shipped six in ruleprobe/detectors/common.yaml use ten of them - tool, arg, command, git, env, text, kind, change, any and all - because that is what those six observables need; message, order, absent and not are exercised by the examples on this page and in tests/, not by a shipped detector.

Three rules about the format worth knowing before you hit them. Every key inside one command block is read against the same pipeline segment, so two constraints on one command belong in one block rather than in an all of two. A session matcher may only be the whole of a session detector's when, because a hit it produces is not a hit on an event in hand. And a list is always alternatives: regex, contains and path_glob hold when any one of their patterns does. contains is a substring, so contains: no-verify finds the token --no-verify; path_glob is a path, so * stops at a /, ** crosses one, and src/*.py matches the absolute path a transcript actually carries.

The format is a YAML subset, and JSON is the same thing. YAML is not in the standard library and this package takes no dependencies, so ruleprobe/declarative.py implements the subset a detector needs - block and flow mappings and sequences, scalars, quoted strings, comments - and refuses everything else by name and line: anchors, aliases, tags, block scalars, directives, and more than one document in a file. A file named .json is read by the standard library's JSON parser into exactly the same objects, so a generator can write JSON and a person can write YAML.

A malformed entry is a finding, not a crash. It is printed with its file, its line and its reason, that entry is skipped, and every other detector in the file still runs.

What it actually covers

Be clear-eyed about the scope, because the name promises more than version 0.1 delivers.

Six detectors ship with the package, and they are the generic ones: reading a whole file into the context window, an unfiltered find, a commit or push that walks past the repository's hooks, a secret-shaped string written to a file, a context compaction, and a model change mid-session. They are in the package because they mean the same thing in every repository, and none of them needs to know what your rules say.

Your own rules take a detector you write, as data in the declarative format above or, when the shape is past what a matcher can say, as Python:

from ruleprobe import DEFAULT, Detector, iter_sessions, measure, report

def sudo_install(events, ctx):
    return [(p.turn, p.id) for p in ctx.bash
            if any(seg[:2] == ["sudo", "pip"] for pipe in p.pipelines for seg in pipe)]

DEFAULT.add(Detector("house-style/sudo-install", "house-style", "bash", sudo_install))
print(report([measure(s) for s in iter_sessions(since=30)]))

The two are the same engine: ruleprobe/detectors/common.yaml is the shipped six written as data, and a test asserts it produces hit-for-hit what the Python in ruleprobe/detectors/common.py produces over the corpus. Python remains the escape hatch, and the seam a third compiler plugs into is still register_compiler and from_spec.

What a matcher cannot say. It reads one event, or one of the three session shapes, and nothing else. There is no arithmetic, no counting ("more than three reads in a turn"), no reading a tool's result, no comparing one argument with another, and no state carried across turns beyond order, absent and change. The command matcher inherits every known miss of the shell parse in ruleprobe/shell.py - a command inside a substitution is invisible, and so is a variable's value. A rule whose shape needs any of that is a Python detector, and the report will not pretend otherwise.

A rule file is one rule. Binding is per file, not per heading: a CLAUDE.md holding twelve rules is one entry in the coverage block, not twelve. Splitting rules into files is what makes the unmeasured list mean anything.

Detector validity is measured, and the measurement is small. Every detector is scored against a hand-labelled corpus that ships with the package - see How good are the detectors? below - but that corpus is synthetic and it is six sessions, so it catches a detector that is wrong about a shape it was shown and says nothing about a shape nobody thought of. The detectors deliberately under-count: a missed hit is a quieter report, a false hit is a wrong one.

What a count is not. A detector fires on a shape in a transcript, not on an intention. whole-file-cat firing 89 times above does not prove the agent wasted context; it proves it read 89 files whole, which is a fact worth having and an argument worth starting.

How good are the detectors?

A hit rate is a rate of the detector until somebody says what the detector should have found. So a labelled corpus ships inside the package, at ruleprobe/corpus/: six synthetic sessions in both transcript shapes, every interesting event labelled by hand with the detectors that ought to fire on it, and a deliberate near-miss beside each one - a cat of a line range, a find narrowed by -name, a git push after the gate ran, a heredoc with rm -rf in its body as text rather than as a command.

ruleprobe corpus

detector                                pos  neg   tp   fp   fn   prec  recall     f1  note
-------------------------------------------------------------------------------------------
cache-hygiene/compact                     5    6    5    0    0   1.00    1.00   1.00
cache-hygiene/model-switch                5   10    5    0    0   1.00    1.00   1.00
secrets/secret-in-write                   6    6    6    0    0   1.00    1.00   1.00
transcript-hygiene/unfiltered-find        5    8    5    0    0   1.00    1.00   1.00
transcript-hygiene/whole-file-cat         5    6    5    0    0   1.00    1.00   1.00
verification/no-verify                    6    6    6    0    0   1.00    1.00   1.00
-------------------------------------------------------------------------------------------
total                                    32   42   32    0    0   1.00    1.00   1.00  floor 0.90

pos and neg are what the labels asked for; tp, fp and fn are what happened. ruleprobe corpus --floor 0.9 exits non-zero when a scored detector falls under the floor, and CI in this repository runs exactly that. It is a gate on the repository, not on a run: nothing in ruleprobe report reads the floor, and no report of yours will ever fail because a detector scored badly. --json prints the same numbers as data.

ruleprobe report --validity puts each detector's p= and r= beside its row. It is off by default because the report is meant to be read in a minute and an eight-column table is not, and because the same two numbers apply to every run - they belong to the detector, not to your transcripts.

Read the number for what it is. The corpus is synthetic and hand-labelled: no real transcript content, no home paths, no personal names. A 1.00 says the detector is right about the shapes somebody thought to write down, which is a weaker claim than it looks - the false positives a detector meets in the wild are the ones nobody anticipated. It is a floor under an obvious mistake and a place to put the next surprising transcript, not a measured field accuracy. Growing it is the cheapest contribution this repository takes: add a session under ruleprobe/corpus/sessions/, label it in ruleprobe/corpus/labels.yaml, and the table above moves.

A detector of your own scores itself. Rather than a corpus, a declarative detector may carry an examples: block of minimal cases, and ruleprobe corpus --rules ./docs/rules scores those:

- id: house-style/sudo-install
  rule: house-style
  event: tool_use
  when:
    command: {starts_with: [sudo, pip]}
  examples:
    fire:
      - bash: sudo pip install ruff
    skip:
      - bash: uv pip install ruff
        note: the tool the rule asks for

fire is a list of cases the detector should fire on and skip a list it should not. A case is bash: <command>, or event: <one event>, or events: [...] for a session detector, with an optional note. Nothing runs them at report time. A detector with no examples and no corpus label prints no examples rather than a number, and the floor steps over it: an unmeasured detector is a gap to see, not a failure to fix.

How it is put together

• ruleprobe/events.py - the event schema every reader emits: assistant_text, tool_use, tool_result, user_prompt, compact.
• ruleprobe/readers/ - one module per runtime, turning a transcript into that schema. Claude Code and Codex today; a reader is ROOT, transcripts() and read().
• ruleprobe/shell.py - the Bash decomposition every shell detector shares. Compounds, pipelines, heredocs, substitutions and continuations, parsed once per command.
• ruleprobe/registry.py - Detector, Registry, run(). Third-party detectors arrive through the ruleprobe.detectors entry point group or through Registry.add.
• ruleprobe/declarative.py - the YAML subset and the front-matter split, with a line number on every refusal.
• ruleprobe/matchers.py - one entry compiled into the same Detector a Python one builds.
• ruleprobe/rules.py - where detector files live, and which rule files nothing measures.
• ruleprobe/report.py - rows in, text out. A row is a small dict, so a report can be taken over rows you stored months ago rather than over transcripts you still have.

The public API is six names:

iter_sessions(root=None, runtime="auto", since=None, errors=None)  # -> Session(.id .repo .events)
run(events, stances=None, *, registry=DEFAULT, strict=False, errors=None)
Registry.add(Detector(id, rule, event, fn, gate=None))
Registry.from_entry_points("ruleprobe.detectors")
report(rows, by="rule", min_sessions=20, promote_share=0.30)
report_data(rows, by="rule", ...)                       # the same numbers as a dict; --json prints it
validity(registry=DEFAULT, directory=None)              # -> {detector_id: Score(.precision .recall .f1)}

plus two for the declarative half:

load_bundle(paths=None, rules_dir=None, cwd=None, config=True)  # -> Bundle(.detectors .rules .findings)
compile_detector(spec, path="<spec>", lines=None)               # -> Detector

Development

git clone https://github.com/JakeSelby/ruleprobe && cd ruleprobe
python3 -m unittest discover -s tests
uv run --python 3.9 python -m unittest discover -s tests   # the floor the package claims
python3 -m compileall ruleprobe
python3 -m ruleprobe report --root tests/fixtures
python3 -m ruleprobe report --root docs --rules docs/rules
python3 -m ruleprobe corpus --floor 0.9                    # the corpus gate CI runs

Origins and neighbours

The engine was carved out of agent-harness, where it grew as a hook that measured that project's own always-loaded rules; the detectors that were about agent-harness's rules stayed there, and the rule-agnostic half is this package. The nearest neighbour is Burnd, which also reads Claude Code transcripts locally, for token spend rather than for rule compliance.

MIT licensed.