agent-steward 0.17.0

A quality gate and spend auditor for your AI agent fleet.

agent-steward

A quality gate and spend auditor for your AI agent fleet.

You run AI agents that work for hours on your behalf — extracting data, writing code, building knowledge bases. Two questions keep nagging you: "Is what they produced actually correct?" and "Am I burning expensive model credits on work a cheap model could do?"

agent-steward answers both, automatically, after every agent session. It checks your agents' output against your own rules, shows you only the handful of items that truly need human judgment, and tells you which tasks deserve an expensive model and which don't.

It never edits your files, never sends your data anywhere, and can be removed in 30 seconds.

---

Who is this for — find your level

You use Claude Code, Cowork, or a similar terminal-based AI agent, and your agent's work lands as files in a repo (notes, data, code, knowledge bases — if the output is files, it's checkable; if it isn't files, this tool has nothing to check). Three levels. Each level's commands are written so an agent can run them for you — paste this README at it.

Level 1 — you just run sessions (this is the Quickstart above)

No pipelines, no automation, no logging discipline. You get both halves on day one:

1. Rule check, every session. The hook runs your manifest after each session; new violations are fed back to the agent to self-repair. Your report shows the Rule check summary and the authorize-fixes-per-row table (below).
2. Measured spend + room-to-move evidence. steward ingest-usage reads the harness's own transcripts — model, tokens, per session, zero manual logging. Your report shows Where the money goes by tier. To judge whether your model choice has downgrade room at this level: run your usual work on a cheaper model for a few days and watch the Rule check section — if violations don't rise while the tier column shifts down, the expensive default was habit, not need. (Evidence-based but manual at L1; L2 automates exactly this judgment.)

What your L1 report contains:

## Rule check — your-project
- 12 rules checked: 10 pass, 2 warn; 23 open findings

### Open findings by category — authorize fixes per row
| category        | findings | how it gets fixed                        | who fixes it |
| rule conflicts  | 1        | two of your rules disagree — steward     | human        |
|                 |          | names both sides; only you can decide    |              |
| schema-floor    | 15       | fill missing fields from the source doc  | agent        |
| code lint       | 7        | ruff --fix + worker self-repair hook     | script       |

## Where the money goes (by tier)
| tier  | weight | runs | tokens    | % volume | % cost |
| cheap | 1      | 6    | 619,000   | 7.4%     | 0.9%   |
| mid   | 8      | 45   | 7,530,226 | 90.1%    | 91.2%  |
| top   | 25     | 7    | 208,000   | 2.5%     | 7.9%   |

The rule conflicts row is the one category no machine may resolve — two of your own written rules disagree (e.g. rule A allows origin: synthesis, rule B doesn't). steward tracks it, names both sides, and holds it for human review; everything else states how it gets fixed so you can authorize per row ("fix everything marked agent/script").

Level 2 — you have repeating jobs (a scheduled run, a daily command)

Everything in L1, plus automated detection of "could this job run a tier lower at the same correctness?" — and the reverse: jobs that keep failing on the cheap tier get promotion proposals.

Enable it (once):

steward allocate rubric                  # your agent rates each job on 4 published axes
steward allocate init --axes axes.yaml   # deterministic matrix -> .allocation.yaml
# tag each job's prompt with [task=daily-ingest]  <- one word, that's the whole protocol

From then on, before a run of a canary-enabled job: steward canary --task daily-ingest — exit 0 means "also run this once on the tier below and compare" (5% sampling by default, deterministic, never below the task's floor). Record the comparison on the shadow entry: steward log-task --task daily-ingest --tier cheap --canary shadow --pair run-12 --quality same.

How "same correctness" is judged: correctness = your rules, not a benchmark. A shadow run counts as equal if (a) it passes the same deterministic checks the primary passed, and (b) you (or your collecting agent) mark its --quality same against the primary at review time. Enough judged runs at parity → a demote proposal with the evidence attached; one measured quality gap vetoes the demote even if failure rates look clean. Cheapest at the same quality, never cheapest at any quality.

What appears in your report that L1 doesn't have — the up/down evidence:

## What needs you
- tier change proposed: condense mid → cheap — 12 of 12 shadow runs (5%
  sampling) passed the same checks at equal quality (see "How 'same
  correctness' is judged") — apply: steward allocate tune --apply --only condense

## Escalations — where a tier proved too low
| task    | from  | redone at | n | why                                          |
| extract | cheap | mid       | 3 | output failed your written rules at this tier |

## Canary — where a tier may be too high
| task     | shadow runs | judged | quality parity | shadow tokens |
| condense | 12          | 12     | 1.0            | 240,000       |

Nothing applies itself: steward allocate tune prints proposals; you accept one with --apply --only <task>.

Level 3 — you run fleets (a dispatcher spawning parallel workers)

Everything in L2, plus the parts that only matter when many agents work at once and the leftovers exceed what you'd read raw:

• Provenance stamps — steward stamp <file> --produced-by <model> --task <t> marks who made each artifact; the allocation_compliance probe audits stamps against the tier table (transition-aware: yesterday's compliant work never becomes retroactively guilty).
• The attention queue — steward route --manifest m.yaml --judge sorts all unresolved findings by impact × judgment-worthiness (the judge runs on your existing claude login, no API key); you work it top-down with steward approve <id> --verdict worth|not-worth. Your verdicts become M4 — the tool measuring its own sorting.
• Verdict memory — steward distill --path <your adjudication log> --where verdict=reject clusters recurring decisions: three same-shaped rejections are one rule waiting to be written (--emit-rubric renders them ready to paste).

Report additions at L3: Attention economics (M1: top-tier tokens per accepted decision; M4: residue precision) and per-rule feedback ("this rule's findings keep being judged not-worth — revisit it").

The problem in one paragraph

When one AI agent does one task, you review it yourself. When a fleet of agents does hundreds of tasks per day, you can't. So either you review everything (you become the bottleneck), or you review nothing (silent errors pile up), or you cap spending with a blunt dollar limit (Tesla-style $200/week caps — which can't tell money well spent from money wasted). agent-steward is the fourth option: machines check what machines can check, you judge only what needs judgment, and every dollar gets a task-level justification.

Quickstart (15 minutes, once per project)

# 1. Install (isolated, doesn't touch your other Python stuff)
pipx install agent-steward

# 2. Draft your project's rules — the agent already working in your
#    project reads the printed rubric and fills in the skeleton from
#    YOUR docs (CLAUDE.md, schema files); you review, you don't write
cd your-project
steward init --out steward.yaml

# 3. Get your "before" numbers (observation only, nothing is blocked)
steward baseline --manifest steward.yaml

# 4. Optional: make it run automatically after every agent session
steward install-hook --manifest steward.yaml

# Lost? Bare `steward` always tells you where you are and what's next.
steward

The three decisions only you can make (everything else is drafted for you)

1. "Which tasks must always use the top model (or a human)?" Some work is too important to delegate down — final decisions, anything touching money, anything irreversible. In the drafted allocation table these are the floor: lines — review them; tuning can never demote below a floor. Example: "final admission decisions and deep reports stay on the top model."

2. "May a cheap model pre-sort the leftovers, and through what credential?" One optional feature (steward route --judge) scores flagged items for judgment-worthiness so you read the queue in the right order. It sends violation excerpts (never whole files) either through the claude CLI login you already pay for, or an ANTHROPIC_API_KEY you set. Don't enable it? The deterministic ranking still works.

3. "Warn or block?" When a rule is violated: just report it (warn), or stop the pipeline (fail)? Start with warn for everything — run in observation mode, see the false alarms, tune, and only then promote your most trusted rules. The drafted manifest defaults every probe to warn for exactly this reason.

What actually gets added to your project

File	What it is	Who writes it
steward.yaml (name it anything)	Your project's rules, as executable checks	Drafted via init's rubric, you approve
.allocation.yaml	Which task types use which model tier	Generated by allocate init from agent-rated axes, you approve floors
One hook entry in .claude/settings.json	"Run steward when the agent finishes"	install-hook, only with your OK

Everything steward produces (state, spend ledger, attention queue, fixes scoreboard) lives in .steward/ — its own folder. Your prompts, your scripts, your code: untouched.

If your project dispatches worker agents, teach the dispatcher two habits (both are one-liners in your project's CLAUDE.md): log each task to the ledger (steward log-task ...), and before dispatching a canary-enabled task, ask steward canary --task extract — exit 0 means "also shadow-run this one a tier lower and compare." No dispatcher? The verification half works on its own.

How it works (60-second version)

Checking (the quality gate). After each agent session, steward scans what changed against your rules — schema fields present, confidence caps respected, code passes lint, files that must exist do exist. Hard rule violations are listed instantly and for free (no LLM involved). Borderline items optionally go to a cheap model for a first opinion. What survives both filters — typically 2-5 items, ranked by how much they matter — is the only thing you look at.

Allocating (the spend audit). Every task gets logged: what type of work, which model tier, roughly how much it cost. Over time steward learns where you're overpaying: if a task type running on the expensive model would pass all your quality checks on a cheaper one (it tests this on a small sample — a "canary"), it proposes a downgrade. If a cheap model keeps failing your checks, it proposes an upgrade. You approve every change; steward never reroutes silently.

Learning (the case file). When you overrule or confirm a flagged item, your reasoning is recorded. Recurring decisions get proposed as new automatic rules — so the same question never reaches your desk twice.

Reading your first report

After each run you get a short terminal summary, and the full report is saved to the run's output folder; steward report gives the cumulative view whose first section is literally titled "What needs you" — if it's empty, you're done for the day. Here's a real-shaped example, annotated:

## What needs you                          ← most days, the only section you read
- tier change proposed: condense mid → cheap (canaries passed at quality
  parity) — apply: steward allocate tune --apply --only condense
- queue top: two knowledge entries contradict each other (affects 2
  published conclusions) — decide which stands, or keep both flagged

## Rule check — my-research-project
- 12 rules checked: 10 pass, 2 warn; 23 open findings

### Open findings by category — authorize fixes per row
| category        | findings | how it gets fixed                      | who fixes it |
|-----------------|----------|----------------------------------------|--------------|
| code lint       | 15       | ruff --fix + the worker self-repair    | script       |
|                 |          | hook; never reaches your queue         |              |
| tier compliance | 8        | remind the dispatcher of cheap-first,  | human        |
|                 |          | or re-assess the tier via tune         |              |

## Where the money goes (by tier)
| tier  | weight | runs | tokens    | % volume | % cost |
|-------|--------|------|-----------|----------|--------|
| cheap | 1      | 6    | 619,000   | 7.4%     | 0.9%   |
| mid   | 8      | 45   | 7,530,226 | 90.1%    | 91.2%  |
| top   | 25     | 7    | 208,000   | 2.5%     | 7.9%   |   ← GOOD: the expensive
                                                              model only judges

## Escalations — where a tier proved too low
| task    | from  | redone at | n | why                                     |
|---------|-------|-----------|---|-----------------------------------------|
| extract | cheap | mid       | 3 | output failed your written rules at this tier |

## Trend
cost per accepted output: −31% vs baseline · escaped defects: 0.11% → 0.09%
→ the two numbers to watch: cheaper AND not sloppier

How to read it as a habit: read What needs you (usually under 5 items), authorize fix categories by row ("fix everything marked script/agent"), glance at the trend once a week. Total: 2-3 minutes per session instead of re-reviewing everything your agents did.

Safety promises (structural, not pinky-swears)

1. Read-only on your project. steward writes only inside .steward/. There is no code path that deletes or edits your files.
2. Offline by default. The rule check makes zero network calls. The only model touch is the optional judge — violation excerpts only (never whole files), through your own logged-in claude CLI or your own API key (hard-whitelisted to api.anthropic.com), and it fails open to the deterministic ranking.
3. Auditable. The engine is a few hundred lines of Python with one dependency. You can read all of it. Prefer paranoia? Copy the single file into your repo and skip the package manager entirely.
4. Fail-open. If steward crashes, your pipeline runs exactly as before — you just don't get a report that round. A watchdog must never take down what it watches.
5. 30-second uninstall. Delete .steward/, remove one hook line. No migration, no residue.

FAQ

Does my data leave my machine? No, unless you enable the judge — and then only violation excerpts, through your own login or key, to the model you chose.

Will it slow my agents down? No. It runs after sessions (seconds, in batch), not between your agent and the model.

What if it flags too much? That's expected in week one. Stay in warn mode, tune the two YAML files (they're short and human-readable), and watch the false-alarm rate drop. Our own first deployment went from 1,270 flags to 251 real ones by fixing two config lines.

Is this a model router / API gateway? No. It never sits between you and the model, never proxies traffic, never holds your keys. It's closer to pre-commit or pytest — a checker you invoke, not infrastructure you live inside.

Real numbers, real projects

Measured on the author's own fleet (a 3,000-node research graph, a personal knowledge system, a trading-ops repo, a docs-only design project):

• 68% estimated spend saved vs running everything on the top model — with the quality guard flat (escaped-defect rate 0.0009, unchanged)
• 289 flagged items → 5 decisions: the queue + judge compressed a day's residue into what actually needed a human; residue precision (M4) 0.93
• 400+ violations resolved and receipted in the fixes scoreboard — including one rule-doc transcription error that alone caused 241 false positives
• 6 minutes to onboard two projects the tool had never seen, using zero new probe types

Full command list and design details: docs/REFERENCE.md.

Status

v0.16 on PyPI. The verification and allocation loops are complete and battle-tested on four real projects. Single maintainer, maintained as I use it — the tool audits its own example configs, so drift shows up in its own reports. Expect fast iteration.

License & contact

MIT. Use it, fork it, sell it, we don't mind.

Questions, ideas, or want to tell us about your setup? [e-mail]