AgentMeasure Healthcheck (pre-release)

A local check-up report for your coding agent runs. Point it at the logs your agent runtime already writes. It answers three questions with evidence — which records are duplicated, where did the agent pay retries for one logical operation, and where did the same tool fail over and over — and writes a terminal summary plus a local HTML report.

Same honesty rules as the conformance pack: every verdict is OK / FINDING / UNPROVABLE, and UNPROVABLE is a first-class result — when the logs cannot decide, that is disclosed, never zeroed.

Status: engineering preview (v0.3.0). Adapters: Codex rollout logs and Claude Code session logs (--runtime claude; auto-detected when Codex is absent). Claude v1 scope: duplicate / retry / error-run checks fully supported; session token accounting is honestly UNPROVABLE — Claude transcripts carry per-request usage only, no cumulative counter, and per-request sums are never presented as session totals. Distribution name decided: agentmeasure on PyPI (command of the same name). PyPI publish is armed and goes live with tag v0.3.0 once the PyPI account/trusted publisher is configured — until then use the git install.

Install

Note (2026-09-21): PyPI still serves 0.4.0, which crashes on the first command (NameError in _run_check; demo / check / compare all affected). The fix is in 0.4.1 and already on main. Until the 0.4.1 wheel lands on the index, install from git instead (option 3 below) — that path runs clean.

# 1) from PyPI (live since v0.4.0 — see note above; 0.4.1 pending upload)
pipx run agentmeasure demo

# 2) install the command
pipx install agentmeasure

# 3) from git (tracks main)
pipx install "git+https://github.com/roy-tong/AgentMeasure#subdirectory=healthcheck"

# 4) zero install, straight from a repository checkout
python3 healthcheck/agentmeasure demo

Python 3.9+ standard library only; zero runtime dependencies; no network code (a test enforces both).

Quick start

agentmeasure demo      # synthetic session — see it work, no data needed
agentmeasure check     # your agent logs, last 7 days (Codex or Claude Code)
agentmeasure check --all           # every local session
agentmeasure check --runtime claude   # force the Claude Code adapter
agentmeasure check --dir ~/.codex/sessions/2026/09/05
agentmeasure selftest  # adapters + checks on bundled fixtures
agentmeasure validate export.json  # check an export against its schema
agentmeasure share run.json        # PREVIEW the sanitized summary (writes nothing)
agentmeasure share run.json --out summary.md   # export after review
agentmeasure --version

(Without installing, prefix the commands with python3 healthcheck/agentmeasure.)

Options for check / demo: --html PATH (default agentmeasure-report.html), --json PATH (full machine-readable export), --share PATH (sanitized summary, .md or .json), --days N, --save-snapshot [PATH], --no-history. demo accepts the same output paths and --no-history.

Input filters for check and compare:

--since 2026-09-01 --until 2026-09-05   # date range (overrides --days)
--project myrepo                        # keep sessions whose project matches
                                        # (project = cwd basename, case-insensitive)

Outcome-billing verification: prepare → recount → dispute → recovery

The buyer-side bill check for per-resolution vendors (Intercom Fin, Zendesk AI Agents, …). Rules live in the vendor-rules registry (single source of truth, each rule cited and confidence-graded). Four commands, one pipeline:

agentmeasure prepare --export native-export.csv --vendor zendesk --inspect
agentmeasure prepare --export native-export.csv --vendor zendesk --out prepared.csv
#   native Intercom/Zendesk export -> canonical skeleton. Judgement columns
#   stay EMPTY for human review; native signal columns travel as hint_*
#   cells; prov_* cells record where every value came from. No verdicts.

agentmeasure recount --export prepared.csv --vendor zendesk --inspect
agentmeasure recount --export prepared.csv --vendor zendesk --json recount.json
agentmeasure recount --export prepared.csv --vendor zendesk --contract contract.json
#   Tier 1: the vendor's own published rules, line by line. Findings in BOTH
#   directions; absent evidence is UNPROVABLE, never guessed. --contract adds
#   the buyer's outcome standard (e.g. reopen_window_hours: 72) as a separate
#   outcome lane — renewal leverage, never netted into the claim.

agentmeasure dispute --export prepared.csv --vendor zendesk --contract contract.json
#   the negotiable pack: cover letter citing the vendor's own rule per
#   finding + appendix + (optionally) Tier 2 under the AMS-1 standard.

agentmeasure recovery --pack pack/dispute-pack.json --confirmations confirmations.csv
#   what the vendor actually conceded: credited/paid per disputed line, with
#   confirmation dates and evidence refs. Realized value is counted
#   separately from the claim — an argument won is not cash.

agentmeasure recount --list-vendors shows the registry with each rule's source and confidence (P = primary/official, S = secondary, U = unverified). The browser-local recount at the website runs the same Tier 1 with no upload; a cross-language parity test keeps the two from drifting.

The whole story in one artifact

agentmeasure crosscheck --export prepared.csv --vendor zendesk --ledger billing.csv
#   your billing ledger (计费流水) against the export: charges the export
#   doesn't flag, billed flags the ledger doesn't confirm, amounts off the
#   published price. Disagreements are named, never resolved.

agentmeasure verify --export prepared.csv --vendor zendesk --contract contract.json \
    --ledger billing.csv --confirmations confirmations.csv --out vledger
#   the Verified Ledger: Tier 1 + outcome lane + ledger cross-check +
#   realized recovery in ONE document (json + md), each lane labelled
#   claim / leverage / realized and never summed across.

agentmeasure dashboard --pack vledger/verified-ledger.json --out dashboard.html
#   offline single-file HTML: cards, tables, recovery bar. No JS, no
#   network, printable — renders, never recomputes.

agentmeasure rules-diff last-month-registry.json
#   the monthly ritual: diff last period's rules snapshot against the
#   packaged registry. Exit 1 = a material change (trigger, price, reopen
#   rule) — re-verify the period under the new rule and disclose it.

agentmeasure narrative --pack pack/dispute-pack.json --audience vendor \
    --emit-prompt prompt.txt
#   step 1 of two: the drafting prompt, built from the pack. Run YOUR OWN
#   model on it, then:
agentmeasure narrative --pack pack/dispute-pack.json --check-draft draft.txt --out letter.md
#   step 2: every number in the draft is checked against the pack; any
#   invented or rounded number refuses the draft and the rule-based
#   template ships. The model never touches the numbers. No model? The
#   template is the default.

agentmeasure delivery --log delivery.jsonl --log-event \
    --engagement ACME --vendor zendesk --period-index 1 --phase mapping --minutes 45
agentmeasure delivery --log delivery.jsonl
#   appendix-E metrics from logged phases: first look ≤10h, third period
#   ≤1h, repeat-period reuse ≥70%. Use the mapping fingerprint `prepare`
#   prints to track which engagements share the reusable mapping.

Outcome pricing beyond per-resolution support (qualified lead, completed case) is defined — schema first, per the BP — at schemas/outcome-unit.schema.json with worked examples. The outcome engine now ships too: agentmeasure outcomes --units units.jsonl verifies them fail-closed, and the schema covers action and usage-aggregate units — one evidence plane across the BP's three stages.

agentmeasure crm-join --export prepared.csv --crm crm.csv --out joined.csv
agentmeasure outcomes --units outcome-units.jsonl --json outcome-ledger.json
agentmeasure period-compare last-month/verified-ledger.json vledger/verified-ledger.json
agentmeasure template --export --vendor intercom --fingerprint <fp> --contract contract.json
agentmeasure certify --pack vledger/verified-ledger.json --digest-out digest.txt
agentmeasure incrementality --events treatment-holdout.csv

Agent commerce: the receipt where the click disappears

Traditional attribution starts at the click; agent commerce has none. The merchant-side measurement receipt (F2.1-F2.9):

agentmeasure commerce-ledger \
    --events agent-events.jsonl --orders merchant-orders.csv \
    --payments merchant-payments.csv --policy-version 1.0.0
#   metric + value + evidence table; retries reconcile to 1 Operation /
#   2 Attempts / 1 verified order; net GMV is strong; incremental is
#   UNPROVABLE (no holdout, no claim); materiality prints the discrepancy
#   vs the naive dashboard in currency AND percent.

live / replay / synthetic never mix inside one aggregation (error, not a blend); metric names carry their strength prefix and are linted against the evidence that backs them; every input is sha256-anchored. The operating layer on top:

agentmeasure pnl-trees --orders orders.csv --campaigns campaigns.csv   # Organic / Paid, ROAS in Paid only
agentmeasure cm-ledger --orders orders.csv --costs costs.csv --period 2026-09
agentmeasure demand-audit --measurements m.csv --thresholds t.json     # Launch / Watch / Not ready
agentmeasure intent-taxonomy --file clusters.json                      # Operating Cell health
agentmeasure decision-audit --policy policy.json --executions log.jsonl
agentmeasure benchmark-export --rows brand-rows.csv --min-brands 5     # aggregates only, ever

Schema: schemas/commerce-profile.schema.json; canonical retry vector pinned in CI at conformance/vectors/commerce-retry-baseline/.

certify grades the artifact on the AMS-1 adoption ladder (Stage 0..3) and emits the deterministic digest; hand the digest to your own signing tool — this package never runs one. incrementality produces statistics about a difference, never a verdict: the BP's line between rule judgement and causal inference stays visible in the output itself.

Sharing results: preview first, export second

The recommended flow never writes a shareable file until you have read it:

agentmeasure check --json run.json          # the report export embeds the summary
agentmeasure share run.json                 # terminal PREVIEW — nothing is written
agentmeasure share run.json --out summary.md  # export only after you reviewed it

share re-checks the summary against its whitelist before showing or writing anything, so a tampered or future-format export is refused rather than leaked. check --share PATH remains available as the one-step explicit alternative.

Snapshots & compare — did the change help?

The re-run preview: save a snapshot, change something (a prompt, a Skill, a dependency), run the agent again, compare.

python3 healthcheck/agentmeasure check --save-snapshot before.json
# …make your change and run the agent…
python3 healthcheck/agentmeasure check --save-snapshot after.json
python3 healthcheck/agentmeasure compare before.json after.json
# or skip the second snapshot: compare runs a fresh check as side B
python3 healthcheck/agentmeasure compare --all before.json

compare prints metric deltas (failed executions, retry chains, …), check verdict transitions (HC-02 OK → FINDING), and honest caveats: token totals are differenced only when both sides are provable, and version / window / mode mismatches are disclosed because they change what a delta is allowed to mean. Snapshots are local artifacts (they contain project names and session short-ids); the snapshot schema is versioned and validated on load.

Versioned export schemas — how other tools integrate (R8)

Everything a third party needs is a JSON document with a version on it:

document	written by	schema
report export	check --json PATH / demo --json PATH	report-v1
snapshot	--save-snapshot PATH	snapshot-v1 (schema field = 1)
comparison	compare --json PATH	compare-v1

Reference schemas live in schemas/. Compatibility policy: within a schema version, fields are only added — never renamed, retyped, or removed; a consumer that meets an unknown newer version must stop and say so. agentmeasure validate FILE… checks any export against its schema, and examples/track-weekly.py is a complete external consumer: it reads snapshots through the schema only, imports nothing from the package, and refuses future schema versions. MCP-style servers can wrap the same exports later; the exports, not internal modules, are the integration surface.

What it checks

id	check	question it answers	verdict basis
HC-01	Duplicate records	byte-identical lines, repeated call/execution ids, sessions split across files, and the format's by-design dual recording when model-side calls and command events are both present	exact line hashes, id counts
HC-02	Retry amplification	same command failed then re-run: how many executions carried one logical operation	consecutive same-command blocks whose first attempt failed
HC-03	Tool error runs	≥3 back-to-back failures of the same tool with no success between	consecutive failed executions, same tool kind

Plus a coverage overview: sessions, turns, executions by outcome, token consumption (with subset discipline), compactions, sub-agent activity, corrupt lines, and a per-project breakdown (project = cwd basename, canonical deduplicated counts). Every finding carries evidence — file, line number, exit codes — and a concrete next step.

Honest-number rules (the part most tools get wrong)

• Executions are counted from exactly one stream. Codex Desktop can write a tool call as both a response_item record and an item_completed event. Summing both — or grepping the file — double-counts when both are present. HC-01 discloses the observed ratio instead of hiding it.
• Token subsets are never added into totals. cached_input ⊆ input, reasoning_output ⊆ output in Codex logs. The report shows subsets alongside, never inside, totals.
• Token totals use the last cumulative snapshot per session, not sums of per-event deltas (which double-count after context compaction). The aggregate is marked UNPROVABLE unless every logical session has one valid snapshot; missing or malformed snapshots are reported separately.
• Corrupt or truncated input lowers every count. All numbers are lower bounds where lines failed to parse or a file hit the safety cap; the limitation is reported, never dropped.
• UNPROVABLE, not zero. Sessions whose format lacks outcome events make HC-02/HC-03 UNPROVABLE for that part — reported as such.

Statistics definitions (for support and accounting)

term	definition
execution	one CommandExecution / McpToolCall event; its outcome is ok, failed, or unknown
retry chain	a maximal block of consecutive executions of the same command whose first attempt failed; one chain = one logical operation executed N times
resolved chain	a chain whose last attempt succeeded
tool error run	≥3 consecutive failed executions of the same tool kind with no success or unknown in between
token total	last cumulative total_token_usage snapshot per session, summed only when every session has a valid snapshot; otherwise UNPROVABLE
project	basename of the session's recorded working directory (session_meta.cwd); used for the breakdown, --project filtering, and snapshots — never included in the share summary
snapshot	versioned JSON record of one run's window, verdicts, and aggregate counters (schema 1); local artifact
comparison delta	B − A per metric; tokens are differenced only when provable on both sides, and version/window/mode mismatches are disclosed as caveats
demo run	executed on the bundled synthetic session; counted separately (synthetic-demo)
own-data run	executed on real local logs (own-data)
run number	local run history at ~/.agentmeasure/history.jsonl distinguishes first vs repeated runs; delete the file to reset

Privacy

• No network. The package contains no network code; a test enforces this by scanning every module for network imports.
• Raw data stays local. Prompts, message text, and reasoning content are never read into the model (only envelope types, ids, exit codes, durations, and token counters).
• Two artifacts, two levels. The HTML report is personal (local paths and evidence metadata — raw commands are represented by hashes, and the banner says so). The --share summary is built from a fixed whitelist of aggregate counts: no prompts, paths, commands, repo names, or session ids can appear in it, and a planted-string test enforces this.
• Run history (~/.agentmeasure/history.jsonl) stays on the machine and is deletable.

Verified runtime support

runtime	status	verified against
Codex rollout JSONL (Desktop originator)	supported	cli 0.142.3 – 0.153.0; review snapshot 54 local files, 48 sessions, ~83k lines (2026-06-30 → 2026-09-05). Sessions without CommandExecution events (older format) run with HC-02/HC-03 UNPROVABLE, disclosed.
Codex CLI (codex_cli_rs originator)	expected compatible, not yet verified — no local samples	pending real samples
Claude Code (~/.claude/projects)	detected, not yet supported	adapter ships after validation against real samples

New cli versions are handled defensively: unknown event types are accounted (unknown type counts), never silently dropped.

Exit codes

code	meaning
0	the check ran and produced a report (findings or not)
1	selftest fixture mismatch (development only)
2	input/output error: no readable logs in the window, bad --dir/arguments, or an output path cannot be written

Development

python3 -m unittest discover -s healthcheck/tests   # 248 tests
python3 healthcheck/agentmeasure selftest           # fixtures + redaction + compare + schemas
bash healthcheck/scripts/smoke_install.sh           # clean-venv install smoke (packaged artifact)

The install smoke is also a CI job (Python 3.9/3.11): it builds the package, installs it into a fresh venv, and runs selftest, demo, validate, and the external example from the installed artifact — the repository checkout is not used.

Python 3.9+ standard library only. Zero dependencies, by policy: if any step asks you to register, connect to a network, or pay — that is a bug.

Feedback

• Open an issue — include the --json export (it contains local paths; strip them first if the log location is sensitive).
• Email