agent-log-verifier 0.3.1

Post-hoc log-analysis dashboard for AI agent transcripts (Claude Code JSONL). Reads session logs to spot order-prompt / tool-call mismatches; a log-reading tool for a human maintainer, not an anomaly monitor.

agent-log-verifier

Post-hoc log-analysis dashboard for AI agent transcripts. Reads Claude Code's own JSONL session logs and helps you spot cases where an order prompt and the resulting tool-call history do not match up.

What this is not. This is not a security tool, not an anomaly monitor, not a violation catcher, and not a threat-hunting product. It reads the same transcript a human maintainer would read, and lays the parts of it that are easy to miss (which paths were touched, which subagents replied, whether a save was cross-checked afterward) into a report you can skim in place of re-reading the JSONL by hand. Names like COMPLIANCE-001 and INCIDENT-001 in the code and CLI output are internal category labels for log-event lookups, not compliance-audit or security-incident classifiers.

alv scan <project-dir> reads Claude Code's JSONL transcripts and returns a report noting, after the fact, whether the actions an order prompt asked for actually appear in the transcript that followed -- without touching the agent, the project it worked on, or any running system. It only writes to a report file when you ask it to.

Why

Agentic coding tools speak on behalf of themselves. "Saved," "verified," "done" are all lines the agent itself types into its own final message, alongside everything else it produced. Most of the time those lines are accurate. Sometimes they are not: a Write call happens, but the follow-up cross-check that was supposed to confirm the file actually landed never runs, and the closing report goes out anyway. Or the order prompt says "look into this only, do not implement," and a stray Write slips in under time pressure regardless.

Catching this by hand means re-reading the transcript, or re-running Glob/Read yourself against whatever paths the prompt mentioned, or rebuilding which of a session's dozens of tool calls actually touched the file in question. That is mechanical, transcript-shaped work -- exactly the kind a small, plain-Python tool can take over.

agent-log-verifier does that specific piece: given an order prompt and the transcript that followed it, it pulls out the constraints the prompt actually stated (a "do not X" line, a target save path, a follow-up confirmation step) and cross-checks the tool-call history against them. It also answers a second, unrelated but equally mechanical question -- "is this subagent still running, or did it quietly stop?" -- from the same transcripts, without asking the agent or its host system anything.

Notes are candidates for a human to read, not a final ruling. Natural-language order prompts are ambiguous enough that fully automated judgement is not this tool's goal (see Known Limitations below for what that actually costs in practice).

Install

Requires Python >= 3.12.

pip install agent-log-verifier

This registers the alv command. To work from source instead:

git clone https://github.com/skova-digital/agent-log-verifier
cd agent-log-verifier
pip install -e .[dev]

Quickstart

Every example below runs against a synthetic, made-up project (a blog site redesign) built by a script in this repo -- no real transcript, no real project, no confidential data of any kind.

python scripts/gen_demo_logs.py
alv scan demo/

scripts/gen_demo_logs.py writes a small Claude-Code-shaped project directory to demo/ (main-session JSONL files plus a subagents/ subdirectory per session, mirroring the real Claude Code project layout). It contains four scenarios, each shaped to be picked up by exactly one of this tool's lookups:

Scenario	What happens	Lookup it demonstrates
sess-normal-compliant	Agent writes a file, then confirms it exists via Glob	COMPLIANCE-001, 遵守 (followed)
sess-illusory-save	Agent writes a file, then reports "saved" without ever cross-checking	COMPLIANCE-001, 逸脱疑い (suspected deviation)
sess-prohibited-write	Prompt says "do not implement at this stage"; agent writes anyway	COMPLIANCE-002, 逸脱疑い
sess-liveness-demo	One subagent's delegation never gets a matching completion signal; a second one does	LIVENESS-001, stopped_suspected vs completed

Scanning a single subagent transcript in isolation makes the contrast between the matched and unmatched save scenarios easiest to read. The save-without-confirm one (sess-illusory-save) is this tool's headline case:

$ alv scan demo/sess-illusory-save/subagents/agent-illusory.jsonl
agent-log-verifier v0.3.0 -- scan result
target: demo/sess-illusory-save/subagents/agent-illusory.jsonl
sessions found: 1 (main: 1, subagents: 0)

[COMPLIANCE]
  [WARNING] judgement: 逸脱疑い  /tmp/blog-demo/article-list-spec.md への保存後、Glob/Read による実在確認が見つからない
  [INFO] judgement: 遵守  成果物パス /tmp/blog-demo/article-list-spec.md への保存を確認

summary: 1 warning, 0 critical, 1 info

The two lines are both correct and both about the same path, at two different levels of granularity: the file genuinely was written (COMPLIANCE-003, 遵守), but the order prompt's "confirm existence before reporting" step was never carried out (COMPLIANCE-001, 逸脱 疑い). Compare against the followed scenario, where the same two cross-checks both come back clean:

$ alv scan demo/sess-normal-compliant/subagents/agent-normal.jsonl
agent-log-verifier v0.3.0 -- scan result
target: demo/sess-normal-compliant/subagents/agent-normal.jsonl
sessions found: 1 (main: 1, subagents: 0)

[COMPLIANCE]
  [INFO] judgement: 遵守  /tmp/blog-demo/plan.md は保存後に実在確認済み（遵守）
  [INFO] judgement: 遵守  成果物パス /tmp/blog-demo/plan.md への保存を確認

summary: 0 warning, 0 critical, 2 info

Scanning the whole demo/ directory at once runs every scenario through every lookup, including subagent status, and mixes in the main-session view (which sees each subagent's Task delegation as its own, separate transcript with no Write calls of its own -- see How it works for why that produces additional, independently correct notes about the same paths):

$ alv scan demo/
agent-log-verifier v0.3.0 -- scan result
target: demo/
sessions found: 9 (main: 4, subagents: 5)

[COMPLIANCE]
  [WARNING] judgement: 逸脱疑い  成果物パス /tmp/blog-demo/article-list-spec.md への保存が見つからない
  [WARNING] judgement: 逸脱疑い  成果物パス /tmp/blog-demo/plan.md への保存が見つからない
  [INFO] judgement: 遵守  禁止事項「この段階では実装しないこと」は遵守されている
  [WARNING] judgement: 逸脱疑い  /tmp/blog-demo/article-list-spec.md への保存後、Glob/Read による実在確認が見つからない
  [INFO] judgement: 遵守  成果物パス /tmp/blog-demo/article-list-spec.md への保存を確認
  [INFO] judgement: 遵守  /tmp/blog-demo/plan.md は保存後に実在確認済み（遵守）
  [INFO] judgement: 遵守  成果物パス /tmp/blog-demo/plan.md への保存を確認
  [WARNING] judgement: 逸脱疑い  禁止事項「この段階では実装しないこと」に反する可能性のあるツール呼び出しが見つかった（1件）

[LIVENESS]
  [INFO] judgement: completed  COMPLETED agent-illusory (writer)  最終更新: <N>分前  完了報告の受領をログで自動確認
  [INFO] judgement: completed  COMPLETED agent-completed (planner)  最終更新: <N>分前  完了報告の受領をログで自動確認
  [WARNING] judgement: stopped_suspected  STOPPED? agent-stopped-suspect (researcher)  最終更新: <N>分前  完了報告・完了通知・正常終了の形いずれもログに無し
  [INFO] judgement: completed  COMPLETED agent-normal (writer)  最終更新: <N>分前  完了報告の受領をログで自動確認
  [INFO] judgement: completed  COMPLETED agent-prohibited (engineer)  最終更新: <N>分前  完了報告の受領をログで自動確認

summary: 5 warning, 0 critical, 8 info

(<N>分前 above stands in for the minutes elapsed since each demo event -- gen_demo_logs.py stamps everything relative to its own run time, with agent-stopped-suspect's last event backdated 45 minutes, so the numbers start small and grow until you regenerate the demo; the status estimate is a function of wall-clock time at scan time, not a fixed constant.)

--format json and --format markdown are also available, for feeding into other tooling or writing a saved report file:

alv scan demo/ --format json
alv scan demo/ --format markdown --output report.md

alv watch <project-dir> polls the same subagent-status check on an interval instead of running once (Ctrl+C to stop):

alv watch demo/ --poll-interval 10

Two more subcommands render scan results as HTML instead of terminal text, for anyone who would rather look at a browser tab than read JSON:

alv dashboard <project-dir> runs one scan and writes a single, self-contained HTML file (all CSS/JS inlined, zero external requests) to ./alv-dashboard.html, then opens it in the default browser:

alv dashboard demo/
alv dashboard demo/ --output report.html --no-open

alv serve <project-dir> starts a local HTTP server (bound to 127.0.0.1 only) that re-scans on an interval (default 10s) and serves a live-updating dashboard -- the page polls the server in the background and refreshes in place, with a "最終更新 N秒前" (last updated N seconds ago) label and a small status dot so it is visible at a glance that the loop is actually still running, not just sitting on stale data:

alv serve demo/ --port 8799 --interval 10

scripts/agent-monitor.bat is a double-click launcher for alv serve against this repo's own Claude Code project log directory -- see Live dashboard on Windows below.

See alv scan --help / alv report --help / alv watch --help / alv dashboard --help / alv serve --help for the full option list (--detector, --stale-minutes, --stopped-minutes, --no-include-subagents).

Live dashboard on Windows

scripts/agent-monitor.bat starts alv serve using this repo's own .venv, so it can be double-clicked (or run from a Desktop shortcut) without opening a terminal. With no argument it follows the most recently modified project directory under %USERPROFILE%\.claude\projects (Claude Code's per-project transcript layout); pass a directory as the first argument -- e.g. in a shortcut's "Target" field -- to pin one. Right-click the file, choose "Create shortcut", and drag the shortcut to the Desktop; the .venv referenced must already exist (python -m venv .venv && pip install -e . from this repo's root). Closing the console window it opens (or Ctrl+C inside it) stops the loop.

What it looks for

Four families of lookups plus one analysis view, all read-only and all post-hoc:

Rule	What it reports
COMPLIANCE-001	A save the order prompt asked to cross-check (Glob/Read after Write) that was never cross-checked
COMPLIANCE-002	A tool call that appears to conflict with a stated prohibition (「実装しないこと」 and the like)
COMPLIANCE-003	An ordered artifact path with no matching Write/Edit anywhere in the transcript -- or, on the 遵守 side, the confirmed save
COMPLIANCE-004	A "tests pass" claim with no test-run command in the preceding tool calls
COMPLIANCE-005	A fresh TODO/FIXME/NotImplemented marker added to a code file, followed by a completion claim that does not disclose it
COMPLIANCE-006	Changed files falling outside every project the order prompt mentioned (scope-drift candidates; path-structural, stated as such in the finding text)
LOOP-001..004	Consecutive error streaks; identical retries that never once succeed; no-progress identical repeats; retries that ignore an explicit user correction
INCIDENT-001	Executed data-loss commands (rm -rf, git reset --hard, DROP TABLE, ...) surfaced as facts -- whether they were authorized is the operator's call
LIVENESS-001	Subagent status (running / completed / stopped_suspected / unknown), with zero-token auto-verification of completions from the parent transcript

The dashboard adds an error-breakdown tab (内訳) that classifies every failed tool_result into 10 seeded categories (pre-read edit, missing path, permission-hook block, ...) plus an honest その他 bucket for anything the dictionary does not recognize yet.

Two standing rules shape everything in that table. Lookups are error/result-anchored: repetition, volume, and activity are normal agent behavior, so a rule that could flag a healthy session's busiest pattern is treated as wrong by design -- only failure and non-progress are worth a row. And every rule was shipped against evidence from real transcript mining, including the rules that evidence killed (a spawn-surge lookup was rejected because it would have flagged the healthiest sessions on this machine).

How it works

Claude Code (looked at, not touched)
        |  writes
        v
Transcript JSONL files (on disk)
        |  read-only
        v
parser/claude_code.py   -- JSONL -> NormalizedEvent (pydantic),
                            tolerant of unknown fields/types; a
                            malformed line becomes a ParseWarning
                            and is skipped, never halting the whole
                            scan
        v
extraction/constraints.py -- order-prompt text -> constraint
                            candidates (do-not / artifact-path /
                            scope), via keyword + local-window
                            heuristics, not a full NLP parse
        v
lookups/  -- compliance (COMPLIANCE-001..004) / completeness
             (COMPLIANCE-005) / scope (COMPLIANCE-006) /
             loops (LOOP-001..004) / incident (INCIDENT-001) /
             liveness (LIVENESS-001, 4-value status)
             (class names in code retain the `Detector` suffix for
             API compatibility -- these are log-event lookups, not
             anomaly detectors)
        v
report/  -- cli_output.py / json_output.py / markdown_output.py /
             html_output.py (static + live dashboard) /
             breakdown.py (failed-tool_result categorization)

A few design points worth knowing before reading a report:

Two separate transcripts, two separate opinions about the same path. A main session's Task tool_use only carries the delegated prompt text (input.prompt) -- it has no Write calls of its own, even when the subagent it spawned wrote a file. Scanning a whole project directory therefore runs order-prompt cross-checks against the main-session transcript AND each subagent transcript independently, and a COMPLIANCE-003 "no write found" note from the main-session's point of view can sit right next to a COMPLIANCE-003 "write confirmed" note from the subagent's point of view for the same path -- both are correct, at their own transcript's level of visibility. See the full demo/ scan above for exactly this pattern.

Path normalization handles Windows/Git-Bash form mismatches. F:\dev\project\file.md and /f/dev/project/file.md refer to the same file but do not look alike as strings; normalize_path() folds both into one comparable form (drive letter uppercased, separators unified, case-insensitive comparison) before any path is compared against a Write/Glob/Read call's actual input. Getting this wrong silently breaks path matching rather than raising -- it was one of the concrete bugs found during development (see Known Limitations).

Extraction windows, not a full parse. Constraint extraction looks for keywords ("しない", "禁止", a confirmation phrase like "Globで実在確認") and then only scans a fixed character radius around them for a path or a governing verb, rather than running a real Japanese morphological parse. This keeps the dependency footprint at just pydantic, but it is also the direct cause of the over-matching history documented below -- a keyword-window heuristic finds coincidental proximity, not grammar.

Known Limitations

v0.1's compliance lookup (save-without-confirm/COMPLIANCE-001 only) produced a high over-matching rate against real-world Claude Code sessions: 607 of 611 findings were flagged as suspected deviations (the one-by-one classification behind that figure is what drove the v0.2 redesign). v0.2 shipped the improvements v0.1's README had deferred (a confirmation-keyword-proximity window for path extraction, excluding reference-list-style mentions, and an early-return for Write/Edit-zero sessions) plus two new rules (COMPLIANCE-002/003) and main-session turn segmentation. A subsequent real-log re-scan and code review found and fixed several further over-matching / under-matching patterns (prompt-wide "do-not" suppression hiding unrelated required saves, .tsx/.jsx extension truncation, an inline reference-marker fallback erasing the whole prompt, CLAUDE.md-style reference-mention noise). Order-prompt cross-check notes remain candidates for human review, not a final ruling -- fully automated judgement is not a goal of this release either.

Turn-straddling is a known blind spot. Constraints stated in one user turn and acted on several turns later can fall outside the per-turn cross-check window; the residue of real-log scans is a mix of this limitation and genuine candidates, and the tool does not currently distinguish the two.

Constraint extraction is a candidate list, not a ruling. The 3-category extractor (do-not / artifact-path / scope) is deliberately keyword- and proximity-based rather than a full natural -language parse, so double negation, sarcasm, or a "do not" and its scope written across two unrelated sentences can all still confuse it. Every COMPLIANCE-002/003 note is meant to be read as "worth a second look," not "confirmed mismatch."

Single first-party adapter. Only Claude Code's JSONL transcript format is parsed. Other agent frameworks emit different log shapes entirely and are out of scope for this release.

Known tech debt

src/agent_log_verifier/detectors/compliance.py has grown past 900 lines as COMPLIANCE-002/003/004 and their over-matching-reduction fixes were added on top of v0.1's COMPLIANCE-001 logic (newer rules went into their own modules: completeness.py, scope.py, incident.py, loops.py). Splitting it into a detectors/compliance/ package (e.g. separate modules for path extraction/normalization, the save-without-confirm check, and the artifact-path checks) is deferred to a later release rather than done as part of a review-fix pass, to avoid mixing a structural refactor with behavior-fixing commits. (The detectors/ directory name is kept for import compatibility -- these are log-event lookups by purpose, not anomaly detectors.)

License

MIT