agent-steward 0.16.0

A quality gate and spend auditor for your AI agent fleet.

agent-steward

A quality gate and spend auditor for your AI agent fleet.

You run AI agents that work for hours on your behalf — extracting data, writing code, building knowledge bases. Two questions keep nagging you: "Is what they produced actually correct?" and "Am I burning expensive model credits on work a cheap model could do?"

agent-steward answers both, automatically, after every agent session. It checks your agents' output against your own rules, shows you only the handful of items that truly need human judgment, and tells you which tasks deserve an expensive model and which don't.

It never edits your files, never sends your data anywhere, and can be removed in 30 seconds.

---

Who is this for

You use Claude Code, Cowork, or a similar terminal-based AI agent. You can run commands in a terminal and you know what a git repo is. That's it — you don't need to be a professional developer.

(This is our starting audience, not the end goal. If you're building for a different setup, open an issue — we want to hear about it.)

The problem in one paragraph

When one AI agent does one task, you review it yourself. When a fleet of agents does hundreds of tasks per day, you can't. So either you review everything (you become the bottleneck), or you review nothing (silent errors pile up), or you cap spending with a blunt dollar limit (Tesla-style $200/week caps — which can't tell money well spent from money wasted). agent-steward is the fourth option: machines check what machines can check, you judge only what needs judgment, and every dollar gets a task-level justification.

Quickstart (15 minutes, once per project)

# 1. Install (isolated, doesn't touch your other Python stuff)
pipx install agent-steward

# 2. Draft your project's rules — the agent already working in your
#    project reads the printed rubric and fills in the skeleton from
#    YOUR docs (CLAUDE.md, schema files); you review, you don't write
cd your-project
steward init --out steward.yaml

# 3. Get your "before" numbers (observation only, nothing is blocked)
steward baseline --manifest steward.yaml

# 4. Optional: make it run automatically after every agent session
steward install-hook --manifest steward.yaml

# Lost? Bare `steward` always tells you where you are and what's next.
steward

The three decisions only you can make (everything else is drafted for you)

1. "Which tasks must always use the top model (or a human)?" Some work is too important to delegate down — final decisions, anything touching money, anything irreversible. In the drafted allocation table these are the floor: lines — review them; tuning can never demote below a floor. Example: "final admission decisions and deep reports stay on the top model."

2. "May a cheap model pre-sort the leftovers, and through what credential?" One optional feature (steward route --judge) scores flagged items for judgment-worthiness so you read the queue in the right order. It sends violation excerpts (never whole files) either through the claude CLI login you already pay for, or an ANTHROPIC_API_KEY you set. Don't enable it? The deterministic ranking still works.

3. "Warn or block?" When a rule is violated: just report it (warn), or stop the pipeline (fail)? Start with warn for everything — run in observation mode, see the false alarms, tune, and only then promote your most trusted rules. The drafted manifest defaults every probe to warn for exactly this reason.

What actually gets added to your project

File	What it is	Who writes it
steward.yaml (name it anything)	Your project's rules, as executable checks	Drafted via init's rubric, you approve
.allocation.yaml	Which task types use which model tier	Generated by allocate init from agent-rated axes, you approve floors
One hook entry in .claude/settings.json	"Run steward when the agent finishes"	install-hook, only with your OK

Everything steward produces (state, spend ledger, attention queue, fixes scoreboard) lives in .steward/ — its own folder. Your prompts, your scripts, your code: untouched.

If your project dispatches worker agents, teach the dispatcher two habits (both are one-liners in your project's CLAUDE.md): log each task to the ledger (steward log-task ...), and before dispatching a canary-enabled task, ask steward canary --task extract — exit 0 means "also shadow-run this one a tier lower and compare." No dispatcher? The verification half works on its own.

How it works (60-second version)

Checking (the quality gate). After each agent session, steward scans what changed against your rules — schema fields present, confidence caps respected, code passes lint, files that must exist do exist. Hard rule violations are listed instantly and for free (no LLM involved). Borderline items optionally go to a cheap model for a first opinion. What survives both filters — typically 2-5 items, ranked by how much they matter — is the only thing you look at.

Allocating (the spend audit). Every task gets logged: what type of work, which model tier, roughly how much it cost. Over time steward learns where you're overpaying: if a task type running on the expensive model would pass all your quality checks on a cheaper one (it tests this on a small sample — a "canary"), it proposes a downgrade. If a cheap model keeps failing your checks, it proposes an upgrade. You approve every change; steward never reroutes silently.

Learning (the case file). When you overrule or confirm a flagged item, your reasoning is recorded. Recurring decisions get proposed as new automatic rules — so the same question never reaches your desk twice.

Reading your first report

After each run you get a short terminal summary, and the full report is saved to the run's output folder; steward report gives the cumulative view whose first section is literally titled "What needs you" — if it's empty, you're done for the day. Here's a real-shaped example, annotated:

── steward report · my-research-project · Jul 12, 21:04 · 4.2s ──

RULE CHECK      2,743 files scanned · 3 new issues since last run
                → "New since last run" is the key phrase: steward never
                  re-shows you old, already-known issues.

AUTO-REVIEW     41 borderline items · 34 cleared · 5 need you · 2 sent back
                → A cheap model pre-screened the gray areas. Most passed.
                  Two were returned to the worker agent with a reason.

NEEDS YOU (3)   ranked by impact — most days you only read this section
  1. Two knowledge entries contradict each other (affects 2 conclusions
     you've published). Decide which stands, or keep both flagged.
  2. One entry claims high confidence from a single source — your rules
     cap that. Lower it, or approve a lookup for a second source.
  3. Worker #2's outputs got 40% shorter this round. Possibly over-
     compressed. Spot-check one?

SPEND           top model: 6% of work volume, 50% of cost  ← this is GOOD:
                the expensive model is doing only judgment work, which is
                exactly what it's for.
                Proposal: simple-extraction canaries passed at quality
                parity → downgrade? Est. saving 18% of mid-tier cost.
                Apply: `steward allocate tune --apply --only extract`

TREND           cost per accepted output: −31% vs your baseline
                escaped defects (errors found late): 0.11% → 0.09%
                → The two numbers to watch: cheaper AND not sloppier.

How to read it as a habit: scan NEEDS YOU (usually under 5 items), approve or reject any PROPOSALS, glance at TREND once a week. Total: 2-3 minutes per session instead of re-reviewing everything your agents did.

Safety promises (structural, not pinky-swears)

1. Read-only on your project. steward writes only inside .steward/. There is no code path that deletes or edits your files.
2. Offline by default. The rule check makes zero network calls. The only model touch is the optional judge — violation excerpts only (never whole files), through your own logged-in claude CLI or your own API key (hard-whitelisted to api.anthropic.com), and it fails open to the deterministic ranking.
3. Auditable. The engine is a few hundred lines of Python with one dependency. You can read all of it. Prefer paranoia? Copy the single file into your repo and skip the package manager entirely.
4. Fail-open. If steward crashes, your pipeline runs exactly as before — you just don't get a report that round. A watchdog must never take down what it watches.
5. 30-second uninstall. Delete .steward/, remove one hook line. No migration, no residue.

FAQ

Does my data leave my machine? No, unless you enable the judge — and then only violation excerpts, through your own login or key, to the model you chose.

Will it slow my agents down? No. It runs after sessions (seconds, in batch), not between your agent and the model.

What if it flags too much? That's expected in week one. Stay in warn mode, tune the two YAML files (they're short and human-readable), and watch the false-alarm rate drop. Our own first deployment went from 1,270 flags to 251 real ones by fixing two config lines.

Is this a model router / API gateway? No. It never sits between you and the model, never proxies traffic, never holds your keys. It's closer to pre-commit or pytest — a checker you invoke, not infrastructure you live inside.

What works today (v0.15)

Everything in the story above is implemented, measured on real projects (a 3,000-node research graph, a trading-ops repo, a personal knowledge system, a docs-only design project), and covered by the test suite. The reference details:

# Run every probe + metric from a manifest against a target project
steward check --manifest examples/your-project.yaml --root /path/to/project

# Same, but report only violations ADDED or RESOLVED since your last check.
# Unchanged noise is suppressed — this is the "3 new issues" experience.
steward check --manifest your.yaml --root /path --diff

# Stamp provenance into an artifact a worker just produced
# (writes produced_by / task / round / stamped_at into its frontmatter)
steward stamp reports/summary.md --produced-by claude-haiku-4-5 --task extract --round 3

# Append one line to the usage ledger (.steward/usage_ledger.jsonl)
steward log-task --task extract --tier mid --model claude-opus-4-8 --est-tokens 1200 --result pass

• 13 probe types, all deterministic and read-only: cmd, jsonl_wellformed, frontmatter_required, single_source_cap, field_value_rule, bash_syntax, csv_required_columns, tsv_wellformed, file_exists, filename_pattern (naming conventions as regex), staleness_flag (files still matching a condition — say verification_status: unverified — past a max age, by a frontmatter date field or file mtime), ref_integrity (frontmatter reference fields — flat lists or nested paths like edges.to — must point at ids that exist in the corpus; dangling reference = violation), and allocation_compliance (checks the model recorded in each artifact's provenance stamp against the tier your .allocation.yaml declares for that task type).
• Rule provenance: every probe in your manifest can carry a source: line naming the authoritative doc and section it enforces — the report shows it, so a flagged violation always points back to your rule, not ours.
• Anti-transcription guard (v0.10): probes that copy numbers out of a rule doc can pin source_file: + source_quote: (verbatim snippets). The engine verifies each quote still exists in the doc — the day the doc changes, you get a SOURCE DRIFT rule-problem instead of a false-positive storm. Born from a real incident: one manifest cap mistyped as 0.6 instead of the doc's 0.9 produced 241 false positives.
• Fixes scoreboard: every check --diff appends violations that stopped appearing to fixes.jsonl; the report shows "Fixed so far" with per-rule counts and examples — the tool's own measurable contribution, not just its complaints.
• Documented exceptions: single_source_cap accepts exempt_field: (e.g. g1_exempt) — a file carrying that field with a written reason is skipped and counted. A reviewable exception beats a rule everyone silently ignores.
• Missing tools are not violations: a cmd probe whose binary is absent (exit 127) reports "missing tool — install it or grant the agent permission to", instead of flooding the queue with environment noise.
• Rule coverage (M5): inventory your project's written rules in a rulebook: section (each entry either covered_by: executable probes, or explicitly judgment_only); the report tells you what fraction of your rulebook is actually enforced by machines, which rules are uncovered (your next probe candidates), and which coverage pointers have gone stale.
• Provenance convention: an artifact is "stamped" when its frontmatter carries produced_by (model/actor), task (task-class id), and optionally round. steward stamp writes these without disturbing anything else in the file; allocation_compliance reads them. Unstamped files are counted, never flagged — adoption is gradual by design.
• State: check results persist to .steward/state.json in the directory you run steward from (never inside a read-only target), which is what powers --diff.

Allocation with zero manual setup (v0.3). Nobody writes the tier table by hand — not even at cold start:

# 1. Print the assessment rubric. Your agent (the LLM session already working
#    in your project) reads your docs and rates every task class on four axes.
steward allocate rubric            # -> agent produces axes.yaml

# 2. The engine maps ratings to tiers with a published, deterministic matrix.
#    The LLM supplies evidence and rationale; the matrix decides the tiers.
steward allocate init --axes axes.yaml     # -> .allocation.yaml

# 3. As the usage ledger accumulates, steward recursively tunes the table:
#    a task class that never fails checks gets proposed one tier down (never
#    below its floor); one that keeps failing gets proposed one tier up.
steward allocate tune              # propose mode: stops and tells you
steward allocate tune --apply      # or set `autotune: auto` in the file
steward allocate tune --apply --only extract   # act on one proposal, keep the rest pending

# 4. Cumulative view, on demand: how much you've saved so far, what the
#    trade-offs were, rule coverage, and any rule problems. Includes a
#    per-day/per-week trend table (granularity auto-picked from the span)
#    and, once tuning has changed a tier, a MEASURED before/after for that
#    task (cost per 1k tokens + escalation rate on both sides of the change).
steward report [--since 2026-07-01] [--until 2026-07-07]

• Reports lead with savings: estimated cost vs an everything-on-top counterfactual and (once tuning kicks in) vs the cold-start table, from est_tokens × declared cost weights. Estimates, clearly labeled — not billing data. Plus CPAU (cost per accepted unit, per task class — every run's cost, failures included, divided by accepted runs).
• Canary shadow runs (v0.8) — cheapest at the same quality, never cheapest at any quality. "Allocated too high" is silent: the expensive model succeeds and nothing tells you a cheaper one would have done the job. For task classes with a canary rate in .allocation.yaml, ask steward canary --task X right before dispatching — deterministic sampling (counted off the ledger, no randomness, replayable) says whether to also shadow-run this one a tier lower (never below the task's floor). Log both halves with --canary primary/shadow --pair <id>, judge the shadow against the primary and record --quality same|worse|better on the shadow entry. Enough judged runs at quality parity → tune proposes the demotion with direct evidence; a measured quality gap vetoes demotion even when the escalation rate looks clean. Shadow runs are bounded exploration cost — excluded from savings math and escalation stats, reported separately.
• The only thing that requires you: rule conflicts the machine cannot decide. check detects rules that govern the same field on overlapping files with incompatible parameters (and coverage pointers that went stale) and names both sides with exact probe ids. Everything else stays automatic.
• Cadence: every check writes a per-run report (seconds, free); steward report gives the cumulative savings view whenever you ask.

Automatic after every agent session (v0.4). steward install-hook --manifest your.yaml registers a Claude Code Stop hook in the target project's .claude/settings.json (created if missing, merged if not — your other settings are untouched, and it refuses readonly targets). From then on, every time your agent finishes a session, steward check --diff --exit-new runs by itself: no new violations → silence (exit 0); new violations → exit 2 with the list on stderr, which Claude Code feeds straight back to the agent for self-repair. Deterministic, and it never forgets to run.

Zero-manual metering (v0.16). Claude Code already records every session's per-message usage — model, input/output/cache tokens, one transcript per spawned worker. steward ingest-usage turns that into MEASURED ledger entries: run it from your project root (it finds the transcripts itself), or point a Stop hook's transcript_path at --transcript. Task attribution is a one-word protocol — put [task=extract] in your dispatch prompts, or pass --task-map 'task_id=regex'; unattributed work still meters by model and tier. Ingested entries carry via: transcript: money views (savings, tier tables, CPAU) count them, while quality loops (tune, canary) keep requiring explicit verdicts — a measurement is not a judgment. Incremental (byte cursor), idempotent, fail-open: the transcript format is Claude Code internal, so anything unreadable is skipped and manual log-task always works. First real run measured 4× more spend than hand-estimates had recorded — mostly the main session's own tokens, which nobody was logging.

Attention routing (v0.9). Deterministic checks leave residue — things a machine can flag but not decide. steward route --manifest your.yaml sorts that residue into an attention queue (.steward/queue.jsonl): score = severity × the probe's manifest-declared risk_weight (your project states what each rule's violations cost; the engine never guesses). Add --judge and a cheap model scores each item's judgment-worthiness — mechanical noise sinks, genuine ambiguity rises. No separate API key needed: if the claude CLI is on your PATH, the judge runs through the terminal login you already pay for; set ANTHROPIC_API_KEY only for headless/CI runs (direct API, hard-whitelisted to api.anthropic.com). Either way it is your own credential, and everything fails open to the deterministic order. You work the queue top-down and grade it as you go: steward approve <id> --verdict worth|not-worth. Those verdicts are M4 (residue precision) — the tool measures its own sorting, and the report shows it next to M1 (top-tier tokens per accepted top-tier run, the price of your attention). The queue proposes an order; it never decides — verdicts stay 100% yours.

The report leads with "What needs you" (v0.11). Everything the machine could not decide, ranked, at the top: rule conflicts and source drift, pending tier-change proposals, rules your own verdicts flagged as noisy, and the top of the attention queue. Everything below that section is evidence, not homework. If this list ever fills with items you judge not-worth, the tool is failing — and M4 will show it.

Verdict memory (v0.11). Adjudication reasons are the most expensive data your pipeline produces. steward distill --path <adjudication log> --where verdict=reject clusters recurring reasons: three rejections with the same shape are one rule waiting to be written. distill --queue does the same for your own worth/not-worth verdicts — rules that keep emitting noise get named, rules that keep catching real problems earn a severity promotion. Candidates only: a human writes the rule; the steward never edits your manifest.

Onboarding a new project (v0.12). pip install the repo and the steward command is on your PATH. Then three steps: steward init --out your-project.yaml prints the manifest-authoring rubric and writes a skeleton — the agent already working in your project fills it in from your own written rules (same zero-manual pattern as allocate rubric); steward baseline runs everything once and seeds the diff state; steward install-hook makes it automatic after every agent session. The distiller closes the loop: steward distill --emit-rubric renders your recurring adjudication reasons as a ready-to-paste L1 judge rubric — your verdicts become your rules.

steward run still works as an alias of check.

Status

Early. The rule check and baseline metrics are battle-tested on two real projects (a research knowledge base with 2,700+ entries and a trading operations repo). Auto-review, ranking, and spend proposals are under active development. Expect sharp edges; expect fast iteration.

License & contact

MIT. Use it, fork it, sell it, we don't mind.

Questions, ideas, or want to tell us about your setup? [e-mail]