AgentMeasure Healthcheck (pre-release)

A local check-up report for your coding agent runs. Point it at the logs your agent runtime already writes. It answers three questions with evidence — which records are duplicated, where did the agent pay retries for one logical operation, and where did the same tool fail over and over — and writes a terminal summary plus a local HTML report.

Same honesty rules as the conformance pack: every verdict is OK / FINDING / UNPROVABLE, and UNPROVABLE is a first-class result — when the logs cannot decide, that is disclosed, never zeroed.

Status: engineering preview (v0.3.0). Adapters: Codex rollout logs and Claude Code session logs (--runtime claude; auto-detected when Codex is absent). Claude v1 scope: duplicate / retry / error-run checks fully supported; session token accounting is honestly UNPROVABLE — Claude transcripts carry per-request usage only, no cumulative counter, and per-request sums are never presented as session totals. Distribution name decided: agentmeasure on PyPI (command of the same name). PyPI publish is armed and goes live with tag v0.3.0 once the PyPI account/trusted publisher is configured — until then use the git install.

Install

# 1) once v0.3.0 is published (planned this week)
pipx install agentmeasure

# 2) today, from git (no account, no PyPI needed)
pipx install "git+https://github.com/roy-tong/AgentMeasure#subdirectory=healthcheck"
# without pipx: python3 -m pip install --user "git+https://github.com/roy-tong/AgentMeasure#subdirectory=healthcheck"

# 3) zero install, straight from a repository checkout
python3 healthcheck/agentmeasure demo

Python 3.9+ standard library only; zero runtime dependencies; no network code (a test enforces both).

Quick start

agentmeasure demo      # synthetic session — see it work, no data needed
agentmeasure check     # your agent logs, last 7 days (Codex or Claude Code)
agentmeasure check --all           # every local session
agentmeasure check --runtime claude   # force the Claude Code adapter
agentmeasure check --dir ~/.codex/sessions/2026/09/05
agentmeasure selftest  # adapters + checks on bundled fixtures
agentmeasure validate export.json  # check an export against its schema
agentmeasure share run.json        # PREVIEW the sanitized summary (writes nothing)
agentmeasure share run.json --out summary.md   # export after review
agentmeasure --version

(Without installing, prefix the commands with python3 healthcheck/agentmeasure.)

Options for check / demo: --html PATH (default agentmeasure-report.html), --json PATH (full machine-readable export), --share PATH (sanitized summary, .md or .json), --days N, --save-snapshot [PATH], --no-history. demo accepts the same output paths and --no-history.

Input filters for check and compare:

--since 2026-09-01 --until 2026-09-05   # date range (overrides --days)
--project myrepo                        # keep sessions whose project matches
                                        # (project = cwd basename, case-insensitive)

Sharing results: preview first, export second

The recommended flow never writes a shareable file until you have read it:

agentmeasure check --json run.json          # the report export embeds the summary
agentmeasure share run.json                 # terminal PREVIEW — nothing is written
agentmeasure share run.json --out summary.md  # export only after you reviewed it

share re-checks the summary against its whitelist before showing or writing anything, so a tampered or future-format export is refused rather than leaked. check --share PATH remains available as the one-step explicit alternative.

Snapshots & compare — did the change help?

The re-run preview: save a snapshot, change something (a prompt, a Skill, a dependency), run the agent again, compare.

python3 healthcheck/agentmeasure check --save-snapshot before.json
# …make your change and run the agent…
python3 healthcheck/agentmeasure check --save-snapshot after.json
python3 healthcheck/agentmeasure compare before.json after.json
# or skip the second snapshot: compare runs a fresh check as side B
python3 healthcheck/agentmeasure compare --all before.json

compare prints metric deltas (failed executions, retry chains, …), check verdict transitions (HC-02 OK → FINDING), and honest caveats: token totals are differenced only when both sides are provable, and version / window / mode mismatches are disclosed because they change what a delta is allowed to mean. Snapshots are local artifacts (they contain project names and session short-ids); the snapshot schema is versioned and validated on load.

Versioned export schemas — how other tools integrate (R8)

Everything a third party needs is a JSON document with a version on it:

Reference schemas live in schemas/. Compatibility policy: within a schema version, fields are only added — never renamed, retyped, or removed; a consumer that meets an unknown newer version must stop and say so. agentmeasure validate FILE… checks any export against its schema, and examples/track-weekly.py is a complete external consumer: it reads snapshots through the schema only, imports nothing from the package, and refuses future schema versions. MCP-style servers can wrap the same exports later; the exports, not internal modules, are the integration surface.

What it checks

Plus a coverage overview: sessions, turns, executions by outcome, token consumption (with subset discipline), compactions, sub-agent activity, corrupt lines, and a per-project breakdown (project = cwd basename, canonical deduplicated counts). Every finding carries evidence — file, line number, exit codes — and a concrete next step.

Honest-number rules (the part most tools get wrong)

• Executions are counted from exactly one stream. Codex Desktop can write a tool call as both a response_item record and an item_completed event. Summing both — or grepping the file — double-counts when both are present. HC-01 discloses the observed ratio instead of hiding it.
• Token subsets are never added into totals. cached_input ⊆ input, reasoning_output ⊆ output in Codex logs. The report shows subsets alongside, never inside, totals.
• Token totals use the last cumulative snapshot per session, not sums of per-event deltas (which double-count after context compaction). The aggregate is marked UNPROVABLE unless every logical session has one valid snapshot; missing or malformed snapshots are reported separately.
• Corrupt or truncated input lowers every count. All numbers are lower bounds where lines failed to parse or a file hit the safety cap; the limitation is reported, never dropped.
• UNPROVABLE, not zero. Sessions whose format lacks outcome events make HC-02/HC-03 UNPROVABLE for that part — reported as such.

Statistics definitions (for support and accounting)

Privacy

• No network. The package contains no network code; a test enforces this by scanning every module for network imports.
• Raw data stays local. Prompts, message text, and reasoning content are never read into the model (only envelope types, ids, exit codes, durations, and token counters).
• Two artifacts, two levels. The HTML report is personal (local paths and evidence metadata — raw commands are represented by hashes, and the banner says so). The --share summary is built from a fixed whitelist of aggregate counts: no prompts, paths, commands, repo names, or session ids can appear in it, and a planted-string test enforces this.
• Run history (~/.agentmeasure/history.jsonl) stays on the machine and is deletable.

Verified runtime support

New cli versions are handled defensively: unknown event types are accounted (unknown type counts), never silently dropped.

Exit codes

Development

python3 -m unittest discover -s healthcheck/tests   # 109 tests
python3 healthcheck/agentmeasure selftest           # fixtures + redaction + compare + schemas
bash healthcheck/scripts/smoke_install.sh           # clean-venv install smoke (packaged artifact)

The install smoke is also a CI job (Python 3.9/3.11): it builds the package, installs it into a fresh venv, and runs selftest, demo, validate, and the external example from the installed artifact — the repository checkout is not used.

Python 3.9+ standard library only. Zero dependencies, by policy: if any step asks you to register, connect to a network, or pay — that is a bug.

Feedback

• Open an issue — include the --json export (it contains local paths; strip them first if the log location is sensitive).
• Email