agent-certify
Structural verification of multiagent systems.
Scan any multiagent system's communication topology. Find structural single points of failure before they're exploited.
Install
pip install agent-certify
The only dependencies are numpy and cryptography. The MCP server, the
OpenTelemetry export and the Cursor, CrewAI and LangGraph parsers need nothing
else — no framework is imported to read its logs (the [mcp], [otel],
[cursor] and [all] extras exist but are empty).
Let your agent scan itself (MCP server)
agent-certify runs as an MCP server, so an agent can check its own structure while it works:
claude mcp add agent-certify -- agent-certify mcp
or in an MCP client's settings:
{
"mcpServers": {
"agent-certify": {"command": "agent-certify", "args": ["mcp"]}
}
}
| Tool | What it returns |
|---|---|
topology_summary |
a plain-language health check, written for the agent to read about itself |
scan_topology |
FI, nodes, links, bridges, articulation points, every node's role (perspective picks the vocabulary) |
blast_radius |
what a compromised or failed node reaches, and whether the system fragments |
check_anomalies |
anomalies against .agent-certify/baseline.json (or baseline_path) |
check_policy |
violations of .agent-certify/policy.json (or policy_path) |
Every tool takes an optional session_path. Without it the server finds the
current session: CLAUDE_CODE_SESSION_ID (Claude Code sets it for the
processes it starts), then the newest session of the working directory's
project. It never falls back to another project's sessions. A real
topology_summary from this repository's own development session:
Your current session has 12 nodes (1 orchestrator, 10 tools, the human) and 11 links. Fragility Index is 1.0 — every connection is a single point of failure with no backup path. The orchestrator (claude) is the sole keystone: if it fails, the system fragments into 11 isolated pieces. Most-used tool: Bash (97 calls, 8 failed). Overall 2.9% of 278 tool calls failed. No anomalies detected against the baseline of 25 sessions. No policy violations. This is the usual shape of a single-agent session — one orchestrator calling each tool directly; redundancy only appears when sub-agents share tools.
The server speaks MCP over stdio (newline-delimited JSON-RPC 2.0, protocol
versions 2024-11-05 to 2025-06-18) without the mcp package. A tool that
cannot run (no baseline yet, unknown node, missing session) returns a normal
result with isError: true and a message saying what to do; protocol errors
are JSON-RPC errors; nothing a request does stops the server.
Tool arguments come from an AI agent, which may be following injected
instructions, so the server treats them as untrusted. Path arguments
(session_path, baseline_path, policy_path) expand ~ but never
environment variables, are resolved (symlinks, ..), and must lie inside
~/.claude/projects or the server's working directory; anything else gets
"Path not allowed". Responses and errors never contain a resolved path, file
contents or exception text: unexpected failures are logged with their
traceback on stderr and reported with a correlation id. Each tool call is
limited to 30 seconds (initialize, ping and tools/list answer directly).
Quick start
from agent_certify import ClaudeCodeParser, AgentFIScanner, BlastRadiusAnalyzer
# Parse a Claude Code session
parser = ClaudeCodeParser()
graph = parser.parse("~/.claude/projects/-home-user-project/session.jsonl")
# Scan structural fragility
scanner = AgentFIScanner(graph)
result = scanner.scan()
print(f"Fragility Index: {result.fi:.4f}")
print(f"Bridges: {len(result.bridges)}")
print(f"Critical nodes: {result.critical_nodes}")
# Blast radius if 'bash' tool is compromised
blast = BlastRadiusAnalyzer(graph)
report = blast.analyze("bash")
print(f"Direct impact: {len(report.direct_impact)} nodes")
print(f"System fragmented: {report.system_fragmented}")
CLI
agent-certify scan ~/.claude/projects/ # every session, plus a merged graph
agent-certify scan ~/.claude/projects/<project>/<session>.jsonl
agent-certify scan <path> --format json|markdown|html [-o FILE]
agent-certify scan <path> --perspective engineering|risk|compliance|monitoring|classic
agent-certify blast ~/.claude/projects/ --node bash
# Baseline and anomaly detection
agent-certify learn ~/.claude/projects/C--dev-myproject/ # writes .agent-certify/baseline.json
agent-certify learn s1.jsonl s2.jsonl s3.jsonl --output ./my-baseline.json
agent-certify scan session.jsonl # compares against ./.agent-certify/baseline.json if present
agent-certify scan session.jsonl --baseline path/to/baseline.json
agent-certify scan session.jsonl --no-baseline
# Tell the baseline what you have decided (survives a re-learn)
agent-certify baseline accept --tool Bash # stop flagging it
agent-certify baseline reject --tool deploy_production # keep flagging it, forever
agent-certify baseline accept --link "agent:Build -> deploy_staging" # that agent only
agent-certify baseline accept --domain api.openai.com # a destination you allow
agent-certify baseline reject --domain "*.cn" # a destination you never allow
agent-certify baseline show # what is on record
# Policy: explicit rules, drafted from the baseline, edited by a human
agent-certify suggest-policy # writes .agent-certify/policy.json
agent-certify scan session.jsonl # also checks ./.agent-certify/policy.json if present
agent-certify scan session.jsonl --policy path/to/policy.json # or --no-policy
# Live monitoring
agent-certify watch ~/.claude/projects/C--dev-myproject/ # newest session in the project
agent-certify watch path/to/session.jsonl --interval 10 --perspective risk
# MCP server logs as a second data source
agent-certify scan session.jsonl --include-mcp
# OpenTelemetry spans (OTLP/JSON)
agent-certify scan session.jsonl --otel # appends to .agent-certify/traces.jsonl
agent-certify scan session.jsonl --otel-stdout | <collector> # spans on stdout, report on stderr
agent-certify watch ... --otel # one span per scan cycle
# Other formats are detected from the content
agent-certify scan path/to/repo/.agent-trace/ # Cursor Agent Trace
agent-certify scan examples/crewai_run.json # CrewAI execution log
agent-certify scan examples/langgraph_trace.jsonl # LangGraph trace
agent-certify learn logs/crew-runs/ # a folder of crew runs
# Structural baseline and drift detection
agent-certify keygen ~/keys/signing.key # mode 600; keep it outside the repository
agent-certify certify <path> --key ~/keys/signing.key -o system.cert.json # valid 90 days (--validity-days)
agent-certify verify system.cert.json <path> --public-key signing.key.pub # exit 1 on drift
# --public-key is required: it is the issuer's key, obtained out of band
# --expect-system-id ID --ignore-expiry --allow-unsigned (skips the signature check; warns)
# In CI: scan exits 2 when a finding reaches --fail-on (critical by default)
agent-certify scan session.jsonl --fail-on warning # or critical (default), never
Node names are matched case-insensitively: --node bash finds tool:Bash.
What it measures
The Fragility Index (FI) measures the proportion of communication links in your multiagent system that have no structural backup path.
- FI = 0: Fully redundant — every path has an alternative
- FI = 1: Fully fragile — every link is a single point of failure
- Bridges: edges whose removal disconnects the system
FI is computed on the agent communication graph using triangle participation analysis. The same structural measure is used by sal-torch on transformer attention graphs (published at PPAM 2026, Springer LNCS).
Node classification and perspectives
Each node is removed in turn and FI recomputed. That gives every node one structural level:
- high: removing this agent/tool raises system FI by ≥ 10% (relative), or the node is an articulation point (removing it disconnects the system)
- medium: raises FI by 2–10%
- low: raises FI by less than 2%, or lowers it
A perspective names the levels for the people reading the report. The measurement is the same under every perspective; only the words (and the colours in the HTML view) change.
| Perspective | high | medium | low | Use it when |
|---|---|---|---|---|
engineering (CLI default) |
KEYSTONE | LOAD-BEARING | PERIPHERAL | designing or refactoring the agent architecture |
risk |
HIGH RISK | MODERATE RISK | LOW RISK | threat modelling, security review |
compliance |
UNRESTRICTED | BOUNDED | LOGGED | audits, access reviews, regulatory reports |
monitoring |
WATCH | TRACK | IGNORE | deciding where to point runtime alerts |
classic |
CRITICAL | BUFFER | IMMUNE | v0.1 labels; the Python API default |
result = AgentFIScanner(graph, perspective="risk").scan()
result.node_classifications # {"claude": "HIGH RISK", ...}
result.node_levels # {"claude": "high", ...}
result.labels("compliance") # same result, other vocabulary
Baselines
agent-certify learn reads N past sessions and records what normal looks like:
- mean ± std of FI, node count, link count, bridge count, sub-agent count and error rate (errors / tool calls), plus FI's range and the most sub-agents seen
- every tool seen, and the fraction of sessions it appears in
- every invocation link seen (
source -> target) - call volume per tool and per link (mean ± std, max, and how many sessions it
was seen in), which is what
call_volume_spikecompares against - every network destination seen (
known_domains) - the tools, links and destinations you have accepted or rejected by hand, which are decisions rather than statistics and so survive a re-learn (see below)
Sub-agent ids are unique per spawn, so links are recorded by role: a sub-agent
counts as agent:<type> (agent:Explore, agent:general-purpose). Standard
deviations are population deviations, so a baseline learned from one session
has std 0.
The baseline is saved to .agent-certify/baseline.json in the current
directory (--output to change). scan picks it up from there automatically.
The baseline is optional: without one, scan behaves exactly as without
anomaly detection.
from agent_certify import AgentFIScanner, AnomalyDetector, BaselineLearner
learner = BaselineLearner()
baseline = learner.learn(["~/.claude/projects/C--dev-myproject/"])
learner.save(baseline)
report = AnomalyDetector(baseline).detect(graph, AgentFIScanner(graph).scan())
report.has_critical
print(report)
Deciding what counts as normal
Learning is a statistic. Whether you accept what it learned is a decision, and
a decision must not be erased by the next learn. Three commands record one:
# A tool you know about. Stops being an anomaly, without re-learning anything.
agent-certify baseline accept --tool Bash
# A tool that must never become normal. It keeps firing even after a re-learn
# that saw it in every single session, and `suggest-policy` will never draft it
# as a REQUIRED tool.
agent-certify baseline reject --tool deploy_production
# Scoped acceptance: this agent may call this tool. Another agent calling the
# same tool still fires.
agent-certify baseline accept --link "agent:Build -> deploy_staging"
# A network destination, as a host or a wildcard over its subdomains.
agent-certify baseline accept --domain api.openai.com
agent-certify baseline reject --domain "*.cn"
agent-certify baseline show # or --format json
All three are stored in baseline.json and carried forward by
agent-certify learn, which recomputes every statistic and leaves the
decisions standing. learn --forget-overrides drops them on purpose.
Scoped acceptance is what makes --link worth having. Approving
agent:Build -> deploy_staging does not approve deploy_staging: if another
caller appears, the tool is flagged again and the message names the callers that
no accepted link covers.
[CRITICAL] New tool never seen: deploy_staging
Baseline: not in 14 known tools
2 call(s) by agent:Test:0 — not covered by an accepted link: agent:Test
A tool can be accepted and then rejected, or the reverse: the newer decision wins. "Permanent" means it survives re-learning, not that you cannot change your mind. Rejection beats a scoped link, so a rejected tool fires whoever calls it.
A link endpoint written with a kind: prefix (tool:Bash, agent:Explore) is
taken as it stands. A bare word is resolved against the baseline — the
orchestrator's id is a bare word (claude) and so is a tool name — and falls
back to a tool, which is the usual case, since the reason to scope a link is
normally a tool that is new.
Rejection is deliberately asymmetric: there is no reject --link. A link you
never want is a policy FORBIDDEN rule, which scan --policy enforces with a
message you write yourself.
--domain records the same kind of decision about a network destination, with
the same precedence; see Network destinations.
Anomaly detection
| Check | Flags | Severity |
|---|---|---|
new_tool |
a tool never seen in the baseline, or one you rejected | critical |
new_link |
an invocation path never seen or accepted (links to a flagged tool are not repeated) | warning |
fi_deviation |
FI more than 2σ from the mean | warning; critical past 3σ when FI rose |
error_rate |
error rate more than 2σ above the mean | warning; critical past 3σ |
sub_agents |
more sub-agents than ever seen, or 2σ above the mean | warning; critical when above the max and past 3σ |
node_count |
node count more than 2σ from the mean | info; warning past 3σ |
call_volume_spike |
a tool, or one link to it, called more than 3σ above the mean | warning; critical past 5σ |
new_domain |
a network destination never seen, and not accepted | critical |
rejected_domain |
a destination matching a rejected pattern | critical |
- critical: something structurally new or a large move toward fragility. Look now.
- warning: a structural change worth a look; often explained by the task.
- info: unusual but benign on its own.
A falling FI means added redundancy (usually sub-agents sharing tools). That is a structural change, not a risk in itself, so it never goes past warning. When the baseline never varied on a metric (std 0), z is undefined: any difference is reported with no z-score at the lower severity.
Call volume
call_volume_spike answers a question the other checks do not: not whether a
tool was used but how much. A tool whose calls jump from 50 to 500 is known,
reaches no new path and may fail no more often than usual — until v0.7.0 it
passed every check, and the injection benchmark measured that as a blind spot
for a release before this closed it.
[CRITICAL] Tool Bash called 487 times (baseline: 50 ± 1, 391.3σ)
called by claude
Volume is learned per tool and per link, and a link spike replaces the tool-level one when it adds something:
[CRITICAL] Link agent:Explore -> tool:Bash called 401 times (baseline: 1 ± 0, …)
agent:Explore alone accounts for this; other callers of tool:Bash
are compared separately
- a tool with one caller is reported as the tool: the link would say the same thing;
- a tool with several callers is reported as the link that spiked, so the message names who is responsible and the other callers are judged against their own links. An agent calling a busy tool its usual number of times is not flagged because somebody else hammered it.
- a tool or link the baseline never saw has no volume to deviate from; that it is
new is
new_tool/new_link's business. - only increases are reported. A quiet session is not a risk.
Per-tool statistics count only the sessions where the tool actually appeared — a tool used rarely but heavily would otherwise have a mean near zero and fire every time it ran at all. With no deviation to compare against (one session, or a count that never varied) twice the mean stands in for 3σ, reported at warning with no z-score.
For a directory, scan lists the anomaly count of each session in the
session table.
How well does detection work?
The checks above are measured, not asserted. agent_certify.benchmark injects
faults whose answer is known into clean sessions and scores what comes back:
python scripts/benchmark.py # synthetic sessions, no logs needed
python scripts/benchmark.py --sessions ~/.claude/projects/C--dev-myproject
python scripts/benchmark.py --format json # for CI
Anomaly injection benchmark — 6 clean session(s) against a baseline of 6
============================================================================================
anomaly type expected trials detected rate severity
--------------------------------------------------------------------------------------------
new_tool new_tool/crit 6 6 100.0% 6 critical
error_spike error_rate/warn 6 6 100.0% 6 critical
new_agent new_link/warn 6 6 100.0% 6 warning
volume_spike call_volume_spike/warn 6 6 100.0% 6 critical
forbidden_tool new_tool/crit 6 6 100.0% 6 critical
foreign_domain new_domain/crit 6 6 100.0% 6 critical
shell_domain (blind spot) 6 0 0.0% not covered by any check
--------------------------------------------------------------------------------------------
detection rate 100.0% (36/36) false negatives 0.0% false positives 0.0% (0/6 clean, 0 anomalies)
blind spots: shell_domain — injected and expected to go unreported, no check covers the fault
| Injection | Fault | Expected |
|---|---|---|
inject_new_tool |
a tool the baseline never saw | new_tool, critical |
inject_error_spike |
existing calls failed until the error rate reaches rate |
error_rate, warning or worse |
inject_new_agent |
an agent of an unseen type calling a known tool | new_link, warning or worse |
inject_volume_spike |
factor× the calls to one tool |
call_volume_spike, warning or worse |
inject_forbidden_tool |
a tool the operator rejected | new_tool, critical |
inject_foreign_domain |
a destination the baseline never saw | new_domain, critical |
inject_shell_domain |
a host named inside a shell command argument | nothing — a measured blind spot |
Three things in that table are worth more than the 100%:
- The volume spike was a measured miss for a release. v0.6.1 reported it as
a blind spot — no check read call volume — and v0.7.0's
call_volume_spikeclosed it. The case stays in the suite, which is the point of measuring a gap instead of omitting it. The two faults added in v0.7.0 change nothing structural at all: the volume spike adds no node, link or failure, and the foreign destination leaves the topology byte-for-byte identical. - One blind spot is open, and it is injected rather than asserted.
Extraction reads hosts from URLs and from fields that name a destination, so
{"command": "curl -sS collector.example.net/x"}reaches no baseline and fires no anomaly.inject_shell_domainbuilds that argument object and hands it to the production extractor; whatever comes back lands in the graph, which today is nothing. It is counted apart from the detection rate so it neither inflates it nor drops out of the report, and it turns into a detection by itself — no edit to the benchmark — the day extraction reads shell commands. "No blind spots left" is a sentence this project will not print unless the measurement says it. - The baseline is learned from sessions that are not the ones tested.
learn_holdout()holds out every other session; a baseline learned from exactly the sessions it is then run against reports nothing on them by construction, and its false positive rate would mean nothing. A baseline that does not fit does produce false positives, and the benchmark says so.
The headline numbers come from synthetic sessions
(agent_certify.benchmark.synthetic), which are deliberately regular. Run
--sessions against your own logs for numbers that describe your system.
Network destinations
An agent graph says which tools ran. It does not say where they went — and if you have to answer where your data is processed, that is the question. An MCP server or HTTP tool repointed at another host changes the answer without changing the topology at all: same nodes, same links, same counts.
So destinations are read from the session's own logs and compared against what the baseline learned:
agent-certify baseline accept --domain api.openai.com # or "*.openai.com" for subdomains
agent-certify baseline reject --domain "*.cn"
agent-certify baseline reject --domain "*.alibaba.com"
agent-certify baseline show # includes a domains section
[CRITICAL] Network destination api.qwen.cn in tool call — not in baseline, matches rejected pattern *.cn
Baseline: rejected pattern *.cn
reached by tool:WebFetch — rejected, so it stays an anomaly across re-learns
What counts as a destination. Logs have no agreed field for this, so extraction is broad but not indiscriminate:
-
any full URL in a tool call's arguments —
https://api.openai.com/v1/chatgivesapi.openai.com(http,https,ws,wss,ftp,ftps); -
a bare host, but only as the value of a field that names one:
url,uri,endpoint,host,hostname,domain,base_url,base_uri,api_url,webhook_url,server,origin,proxy, their plurals, and a dozen more —NETWORK_KEYSinagent_certify/domains.pyis the full list.{"host": "db.internal"}counts; an IP literal counts too.One exception, to keep configuration noise out of the report: a value whose last label is a common file extension (
notes.txt,data.csv,app.log—NOT_HOSTSin the same module) is read as a filename, not a host. Every label on that list is absent from the DNS root, so no reachable name is dropped by it;collector.zip,x.sh,y.so,z.md,a.pyandb.rsare destinations and are reported as such.
A domain-looking word anywhere is not a destination. That distinction is what
keeps this usable: a session that reads a file mentioning example.com, or
greps for alibaba.com, has contacted nobody, and reporting it as egress would
bury the real finding. For the same reason a tool result is read only when
that call's own arguments carried a URL — so a WebFetch records its redirects,
while a Read of a document full of links records nothing.
Patterns, not addresses. Nothing is resolved, in either direction. A pattern is matched against the name the agent used, which is the thing you made a decision about:
| Pattern | Matches | Does not match |
|---|---|---|
api.openai.com |
that host only | other.openai.com |
*.cn |
api.unknown.cn, deep.sub.unknown.cn |
cn, notcn |
*.alibaba.com |
x.alibaba.com, x.y.alibaba.com |
alibaba.com, evil-alibaba.com |
A wildcard covers subdomains only: the bare domain is a destination of its own
and deserves its own decision. .cn is accepted as a spelling of *.cn, and a
whole URL is accepted and reduced to its host. There is no * pattern — a host
that is not in the baseline already fires as new_domain, so deny-by-default is
the behaviour you get without asking for it.
Decisions follow the same rules as tools: stored in baseline.json, carried
forward by learn, last write wins, and reject beats accept — a rejected
pattern fires for a host the baseline learned in every session.
Destinations are read by the Claude Code, CrewAI, LangGraph and MCP-log parsers,
appear in scan --format json as seen_domains, and are listed by
agent-certify learn and baseline show.
Example workflow
# 1. Learn what normal looks like from the last few weeks of sessions
agent-certify learn ~/.claude/projects/C--dev-myproject/
# 2. Scan a new session; anomalies print above the node table
agent-certify scan ~/.claude/projects/C--dev-myproject/<new-session>.jsonl
# 3. Triage with the vocabulary your team uses
agent-certify scan <new-session>.jsonl --perspective monitoring --format html -o review.html
Fragility Index 0.3103 (9/29 links with tri(e)=0; baseline: 0.98 ± 0.09)
⚠ ANOMALIES DETECTED (8):
[CRITICAL] Sub-agents: 3 (baseline: 0.04 ± 0.20, max seen 1)
Types: agent:Explore, agent:general-purpose
[WARNING] New communication path: claude -> agent:Explore
Baseline: not among 35 known links
[WARNING] FI deviation: 0.31 (baseline: 0.98 ± 0.09, z = -7.8)
Sub-agents detected — shared tools add triangles; structural change expected
...
Blast radius
BlastRadiusAnalyzer.analyze(node) reports:
- direct_impact: 1-hop neighbours. A compromised tool can steer its caller through poisoned results, so spread is measured in both directions.
- total_reachable: everything within 3 hops (
hops=to change). - accessible_tools: tools the node can invoke itself or through the agents it spawns (follows call direction).
- remaining_fi / system_fragmented: structure after isolating the node. "Fragmented" means isolation splits the system into more pieces than it had.
Certificates and drift
An AgentCertificate is a signed (Ed25519) structural baseline: the nodes, the
directed invocation links, FI and node roles. graph_hash covers the topology
only, not call counts. verify(graph) names each change: new agent types, new
tool links, links that disappeared, FI movement, role changes. A topology change
can be detected before the behaviour it enables shows up.
Sub-agents are compared by type, not by their per-spawn id: a session that
spawns a fresh Explore agent for the same tools matches a certificate issued
for another Explore agent. A different type is reported as such:
New agent type: agent:Research (not in certified baseline)
Missing agent type: agent:Explore (was in certified baseline)
Certificates issued before v0.3 have no identity field; they keep comparing
agents by id, and their signatures remain valid.
Trust model
A signature proves something only when it is checked against a key you
already trust. verify therefore requires the issuer's public key
(--public-key, or public_key= in Python), obtained through a trusted
channel. The signer_public_key written inside a certificate is informational
only: whoever can edit the certificate can replace it. A certificate fails
verification when it is unsigned, signed by another key, edited after
signing, outside its validity window, or issued for another system
(--expect-system-id). --allow-unsigned skips the signature check
explicitly and prints a warning; use it only for local experiments.
Since v0.5 every certificate carries not_before / not_after (90 days by
default, --validity-days), so an older, more permissive certificate cannot be
replayed indefinitely. Certificates from earlier versions have no validity
window; they still verify, with a warning, and should be re-issued.
The signed bytes are b"agent-certify/cert/v1\x00" followed by the
certificate as JSON with sorted keys, compact separators, ASCII escaping and
fi_score written as a 10-place decimal string; see certificate.py for the
exact rules. Older certificates keep their original encoding, and the format
is part of what is signed, so a certificate cannot be downgraded.
Keys: agent-certify keygen creates the private key with mode 600 and refuses
to overwrite an existing file or follow a symlink. certify warns when the key
is readable by other users. On Windows the file inherits the folder's ACL, so
keep keys in a folder only you can read. Never commit a key; *.key is in
.gitignore.
Policy
Anomalies say this is unusual. A policy says this is not allowed (or this
must be there), whatever the statistics. A session can have both, and scan
reports them separately.
{
"rules": [
{"kind": "forbidden_tool", "target": "deploy_production", "severity": "critical", "message": "No agent may deploy to production"},
{"kind": "forbidden_link", "target": "agent:* → *database*", "severity": "critical", "message": "Sub-agents may not reach the database"},
{"kind": "max_sub_agents", "target": "5", "severity": "warning", "message": "More than 5 sub-agents is unusual"},
{"kind": "required_tool", "target": "Bash", "severity": "warning", "message": "Bash is expected in every session"}
],
"generated_from": "baseline (25 sessions)",
"created_at": "2026-10-27T20:00:00Z"
}
| kind | target | violated when |
|---|---|---|
forbidden_tool |
tool pattern | a matching tool is used |
forbidden_link |
"source → target" (or ->) |
a matching invocation link exists |
max_sub_agents |
integer | more sub-agents than that |
required_tool |
tool pattern | no matching tool is used |
Patterns are shell-style globs (*, ?), case-insensitive, matched against a
node's id (tool:Bash), type (agent:Explore) and name (Bash). agent:*
matches every sub-agent; a bare name matches exactly (deploy does not match
deploy_production; deploy* does).
agent-certify suggest-policy drafts a conservative policy from the baseline:
tools seen in every session become REQUIRED (warning), and the sub-agent count
is capped at the baseline maximum + 1 (warning) and + 3 (critical). FORBIDDEN
rules are an organisational decision, so the draft never contains any; add
them by hand. suggest-policy refuses to overwrite an existing policy file
(which may hold your edits) without --force.
Live monitoring
agent-certify watch follows a session while Claude Code writes it. Given a
project directory it picks the most recently modified session. Every
--interval seconds (default 30) it checks whether the transcript or any of
its sub-agent transcripts grew; if so it re-parses, rescans and prints:
Watching C:\Users\me\.claude\projects\H--agent-certify\c603eabc-….jsonl (every 5s, Ctrl+C to stop)
Baseline: loaded (25 sessions)
Policy: loaded (4 rules)
────────────────────────────────────────────────────────────
[11:35:02] Nodes: 7 Links: 6 FI: 1.0000 Calls: 192
[CRITICAL] POLICY VIOLATION: FORBIDDEN tool used: Edit
[11:35:07] Nodes: 7 Links: 6 FI: 1.0000 Calls: 192
[11:35:27] Nodes: 7 Links: 6 FI: 1.0000 Calls: 196
[11:35:37] Nodes: 7 Links: 6 FI: 1.0000 Calls: 197
[11:35:52] Nodes: 7 Links: 6 FI: 1.0000 Calls: 200
(A real run on this repository's own development session, with a demo policy
that forbids Edit.)
Each anomaly (critical and warning) and each policy violation is printed once,
when it first appears, and again only if its severity changes. A session file
that does not exist yet is waited for; a half-written last line is skipped
until it is complete. It uses plain size polling, no file-system notification
library. --scans N stops after N scans and --duration S after S seconds — the only way to end a watch on a session that has gone quiet; in Python, SessionWatcher(...)
takes on_anomaly / on_policy_violation callbacks and has stop().
MCP server logs
Claude Code also logs every MCP server connection, one JSONL file per server and process start:
| Platform | Location |
|---|---|
| Windows | %LOCALAPPDATA%\claude-cli-nodejs\Cache\<project>\mcp-logs-<server>\*.jsonl |
| Linux | ~/.cache/claude-cli-nodejs/<project>/mcp-logs-<server>/*.jsonl |
| macOS | ~/Library/Caches/claude-cli-nodejs/<project>/mcp-logs-<server>/*.jsonl |
Records are {"timestamp", "sessionId", "cwd", "debug" | "error"}; tool calls
appear as Calling MCP tool: <tool> and failures as Tool '<tool>' failed ….
These logs include calls Claude Code makes on its own, such as the IDE
server's getDiagnostics after an edit, which never appear in the transcript.
--include-mcp (or MCPLogsParser().enrich(graph)) adds the calls whose
sessionId matches the session, as claude → tool:mcp__<server>__<tool>. A
tool the transcript already shows keeps its node, and calls already counted
there are not counted twice. The logs do not say which agent made a call, so
all of them are attributed to the orchestrator. No logs (no MCP servers, or a
platform without the cache) is not an error: the graph is returned unchanged.
OpenTelemetry
--otel turns every scan into one span, agent_certify.scan, with the
structural metrics as attributes (agent_certify.fi, .node_count,
.edge_count, .bridge_count, .sub_agent_count, .anomaly_count, ...).
Anomalies and policy violations become span events (anomaly.new_tool,
policy_violation.forbidden_tool, ...). A critical anomaly or any policy
violation sets the span status to ERROR.
Spans are OTLP/JSON, one {"resourceSpans": [...]} object per line: the
layout the OpenTelemetry Collector's file exporter writes and its
otlpjsonfile receiver reads, so a collector can forward them to Jaeger,
Grafana Tempo, Datadog and so on. All scans of one session share a trace id
derived from the session id, so a watch run appears as one trace with a span
per cycle; a graph with no session id (a Cursor trace that does not carry one)
gets a random trace id per span instead. No OpenTelemetry SDK is needed.
Cursor Agent Trace
Agent Trace (github.com/cursor/agent-trace) records
which code was written by AI and which by humans: files, line ranges and
contributors (human, ai with a model_id, mixed, unknown), usually in
.agent-trace/traces.jsonl. It does not record tool calls, MCP servers or
sub-agents, so the graph is a provenance graph:
- the producing tool (
cursor) is the orchestrator, linked to each model (agent:<model_id>) - each contributor links to the files it wrote (
file:<path>), weighted by lines
That graph is bipartite (contributors on one side, editor and files on the
other) and has no triangles, so FI is always 1.0 for it. The useful views
are blast radius (which files a model touched), policy rules such as
{"kind": "forbidden_link", "target": "agent:* → file:infra/*"} and anomaly
detection (a new model, new files touched).
Formats are detected from the content, not only the extension:
.agent-trace/traces.jsonl is routed to the Cursor parser although it ends in
.jsonl, and a CrewAI log is recognised whether it is .json or .jsonl. The
first 20 records of a file are probed and the first one a parser claims decides;
a file nothing claims is read as a Claude Code transcript, as before. A
directory is scanned for every supported format: Claude Code sessions,
.agent-trace/ folders, *.agent-trace files, and CrewAI or LangGraph logs
(MCP server logs are left out; they enrich sessions via --include-mcp). A
.json file that holds no records of any format — package.json, say — is
ignored.
CrewAI
A CrewAI crew writes its execution log as JSON
(Crew(output_log_file="run.json")), and its event bus is commonly dumped to
JSONL; both are read, as is a single object wrapping the entries
({"events": [...]}). Records are matched on content, since the layout differs
between CrewAI versions: an entry is read when it names an agent
(agent, agent_role, agent_name, role, agent_key) and at least one of a
task, a tool, a crew or a status.
| Log content | Graph |
|---|---|
| the crew | crew (orchestrator) |
inputs on a kickoff |
human → crew |
| each agent role | agent:<role> |
| a task assigned to an agent | crew → agent:<role>, weight = distinct tasks |
| a tool call | agent:<role> → tool:<name> |
Delegate work to coworker / Ask question to coworker |
agent:<from> → agent:<coworker>, besides the tool edge |
an error, or status: failed |
error count on that edge |
Node ids are the roles themselves (agent:Researcher), so an agent is the same
node across runs and a baseline can be learned over a crew's history. A call is
counted once whether the log holds its start, its completion, or both (paired by
tool_call_id/id/run_id, or by agent + tool + task). Delegation is what
gives these graphs structure: crew → Writer, crew → Researcher and
Writer → Researcher close a triangle, so FI falls below 1 and the delegation
links have a backup path while the tool calls do not.
LangGraph
LangGraph runs a state machine of
named nodes. Three trace shapes are read, from .json or .jsonl:
- Event stream —
graph.astream_events(..., version="v2"):metadata.langgraph_nodesays which node a record belongs to andmetadata.langgraph_triggerswhich node handed control to it. - Debug stream —
stream_mode="debug":{"type": "task", "payload": {...}}and itstask_result. - Plain transition records — what a hand-rolled logger writes:
{"node": "agent", "next": "tools", "tool_calls": [...]}.
| Trace content | Graph |
|---|---|
| the compiled graph | graph (orchestrator) |
| the input it was invoked with | human → graph |
| each node that ran | agent:<node> |
| a state transition | agent:<from> → agent:<to>, weight = times control took that path |
__start__ as the source |
graph → agent:<node> |
| a tool invocation | agent:<node> → tool:<name> |
on_tool_error, or an error field |
error count on that edge |
A node is modelled as an agent whose type is its name, so agent:tools is
stable across runs. A node and the tools it calls stay separate: the tools
node of a ReAct graph is agent:tools, the tool it ran is tool:search.
__start__ is the orchestrator, __end__ is dropped (a sentinel, not a
participant), and a node that loops back to itself adds no edge — the repeat
shows in its call count. Consecutive records about the same node are one
activation, not a loop, so they move neither count.
Transitions come from langgraph_triggers/triggers, then from an explicit
next/to, and otherwise from the order the nodes ran in. That last fallback
is a guess: parallel branches interleave, so a trace without triggers can show a
transition the graph definition does not have. Traces from astream_events or
stream_mode="debug" carry triggers and do not rely on it.
How Claude Code sessions map to graphs
| Log content | Graph |
|---|---|
| the main transcript's assistant | claude (orchestrator) |
| prompts typed by the person | human → claude |
tool_use blocks |
caller → tool:<name> (shared across agents) |
Agent / Task calls (or any call with subagent_type) |
caller → agent:<id> |
<session>/subagents/agent-<id>.jsonl |
that agent's own tool calls |
tool_result with is_error: true |
error count on the edge and the tool |
Streamed messages repeat tool_use blocks, so calls are de-duplicated by id.
Claude Code transcripts and MCP logs are read line by line; a Cursor
.agent-trace file of up to 50 MB is read whole (it may be a single JSON
document), larger ones line by line. Lines that cannot be read — malformed,
nested too deeply, or longer than 10 MB — are skipped and counted: the count
appears in the scan report, in JSON (skipped_records), in OTel spans and in
certificates. A warning is printed as soon as one record is skipped — on a
verification tool, an incomplete graph is worth saying out loud however small
the fraction — and the scan report gives that fraction. A directory scan reads
at most 1,000 session files and walks at most 5 folder levels.
A session without sub-agents is a star (one orchestrator, many tools) and has FI = 1 by construction. Structure becomes informative once sub-agents share tools: each shared tool closes a triangle with the orchestrator.
Supported systems
- Claude Code (JSONL logs from ~/.claude/)
- Claude Code MCP server logs (
--include-mcp) - Cursor Agent Trace (code provenance;
.agent-trace/) - CrewAI (execution logs and event bus;
.json/.jsonl) - LangGraph (event stream, debug stream or plain transition records)
- AutoGen (planned)
- OTel traces (planned)
New sources subclass agent_certify.parser.BaseParser and return an AgentGraph.
Links
- sal-torch — structural analysis of AI models
- sal-certify — structural AI certification
- cognitive-engineering.dev
agent-certify is built by Cognitive Engineering (Switzerland). License: Apache 2.0.
Metadata
Release files for agent-certify 0.7.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| agent_certify-0.7.0.tar.gz | 235.3 kB | With provenance | Details for agent_certify-0.7.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| agent_certify-0.7.0-py3-none-any.whl | Python 3 | none | any | Details for agent_certify-0.7.0-py3-none-any.whl |
Total release size: 386.2 kB
Release files / agent_certify-0.7.0.tar.gz
| Download URL | agent_certify-0.7.0.tar.gz |
|---|---|
| Size | 235.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
0f19a44ab584e2ca7f3469b9179bb8afc3285fc552cccc96f173f4beb44654b0
|
|
BLAKE2b-256 checksum How to use checksums |
d6cede01476cff63d906d389684ee4fec8fb0d5337e8963bf0d58f45a1a63018
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.
Transparency logRelease files / agent_certify-0.7.0-py3-none-any.whl
| Download URL | agent_certify-0.7.0-py3-none-any.whl |
|---|---|
| Size | 151.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e74e1a357fb2f72e7bdd4c65d32a833c9dc7dd25bcb77357683e3b7d004f5940
|
|
BLAKE2b-256 checksum How to use checksums |
c558bcc2ef73bfeadf01ce4b83d16b5782e9b9765d34b342d72a930bb9624f12
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.
Transparency logSecurity for agent-certify 0.7.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for agent_certify-0.7.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.