agentnorm

Behavioural monitoring for AI agents. Evaluation grades what an agent said, offline. This watches how it behaves, at runtime, and tells you when a run does not look like the ones before it.

Zero dependencies. No database, no framework, no context propagation.

pip install agentnorm      # zero dependencies

from agentnorm import RunRecorder, Monitor

rec = RunRecorder(agent="triage", version="v3", principal="acme")

with rec.tool_call("search_tickets", {"q": q}, scope="acme") as call:
    rows = search(q)
    call.result_size = len(rows)

monitor = Monitor.fit(history)            # past benign runs
verdict = monitor.score(rec.finish())

if verdict.flagged:
    print(verdict.explain())
    # volume=7.31 (threshold 3.02) [cold start: no history for this agent version]

Instrumenting an agent you already have

Hand over the tool callables you already use; get back callables with the same signatures that record as a side effect. No restructuring, no context propagation.

from agentnorm import JsonlStore, Monitor, Session

store = JsonlStore("history.jsonl")

session = Session(agent="triage", version="v3", principal="acme",
                  scope_of=lambda tool, args, result: args.get("tenant"))
tools = session.wrap({"search_tickets": search_tickets, "export_all": export_all})

# ... run the agent using `tools` exactly as before ...

store.append(session.finish())
verdict = Monitor.fit(store.read()).score(session.finish())

scope_of is what makes cross-tenant detection possible at all - without it agentnorm can see that a call happened but not whose data it touched. Returning None means "unknown", which is treated as in-scope, because a false accusation of cross-tenant access is worse than a miss.

What that gets you

From examples/quickstart.py, fitted on 300 normal runs:

normal run           -> no anomaly

exfiltration attempt -> scope=1.00 (threshold 0.00); novel_tool=1.00 (threshold 0.00);
                        sequence=3.61 (threshold 0.00); volume=7.98 (threshold 2.17)

new agent version    -> no anomaly [cold start: no history for this agent version;
                                    uncalibrated: sequence, novel_tool, rate]

Four detectors fire on the exfiltration attempt and each names the invariant that broke - wrong tenant, unfamiliar tool, unusual path, far too much data. That is the difference between "something is wrong" and a page an engineer can act on at 3am.

The third line is the one most monitoring gets wrong: a version bump is a change-point, not an anomaly. agentnorm reports that it has no history rather than alerting, and names which detectors are consequently unavailable.

It also refuses to pretend about calibration:

warning: calibration set has 75 runs but a 0.0020 quantile needs at least 500;
thresholds fall back to the observed maximum and the true false-positive rate
will exceed the budget

Framework adapters

For frameworks that report tool calls through callbacks rather than direct invocation:

from agentnorm import Session
from agentnorm.adapters.langchain import agentnorm_callback

session = Session(agent="researcher", version="v2", principal="acme")
graph.invoke(state, config={"callbacks": [agentnorm_callback(session)]})

verdict = monitor.score(session.finish())

Works with LangChain and LangGraph. agentnorm still does not depend on either - the base class is imported lazily and the handler works as a plain object without it, because LangChain duck-types handlers in the paths that matter. That is deliberate: a monitoring library that drags in an agent framework is unusable by anyone running a different one, and comparing agents across frameworks on equal footing is half the point.

The adapter is tested with LangChain deliberately not installed - the callback contract is replayed instead - so the zero-dependency guarantee stays testable in CI.

Two behaviours worth knowing, because callback streams are messier than they look:

• Concurrent tools are matched by the framework's run_id. LangGraph runs tools in parallel, so starts and ends interleave and pairing them by order is wrong.
• A call the framework never ends is recorded as failed, not discarded. An agent killed mid-tool leaves an open call, and a run that ends inside a tool is itself a signal. Conversely an end with no start - the handler attached mid-run - is dropped, because a call with no beginning has no duration and no arguments, and inventing them would corrupt the baseline it feeds.

Persistence

Monitor.fit needs history, so a monitor that cannot remember is useless in practice. JsonlStore is the smallest thing that solves it: append-only JSON Lines, standard library only. Append-only is deliberate - behavioural history is evidence, and a store that can be rewritten in place is worth much less during an investigation. A truncated final line from an interrupted write costs one run, not the file.

Store is a protocol, so ClickHouse or Postgres is a drop-in. The reference deployment uses ClickHouse.

Why this exists

Agents are unvalidated models running in production. We have good tooling for grading their outputs before release and almost none for answering the question that matters after release: did this run behave like the others?

That question is closer to fraud detection than to evaluation. It is about the shape of an episode — which tools, in what order, touching whose data, returning how much — not about whether an answer was good.

Five detectors, each owning one failure mode, deliberately not fused into a single score. A fused score says something is wrong without saying what, and "what" decides whether you page someone, revoke a credential, or ignore it.

Detector	Catches
volume	an agent returning far more data than usual — exfiltration-shaped
sequence	a path no planner would take, where every individual call is legitimate
scope	a run entitled to one principal touching another's resources
novel_tool	reaching a tool this agent has never used
rate	an order-of-magnitude change in calls per run

The hard part: cold start

Agent versions change weekly. A detector meeting an entity it has never seen is not an edge case — it is the normal operating condition, and it is where naive monitoring falls apart.

Measured on a live deployment: a suite fitted on one population and scored against a real agent it had not seen alerted on 100% of known-benign runs. Every tool looked novel, because the agent was novel. The detector was reporting "I have not met you" on every run, forever.

The per-detector breakdown is the useful part:

Detector	Alert rate on benign runs from an unseen agent	After
volume (hierarchical)	0.000	0.000
sequence	0.000	0.000
scope	0.000	0.000
novel_tool	1.000	0.000
rate	0.536	suppressed
any	1.000	0.000

The hierarchical model transferred untouched; the set-membership ones had no cold-start behaviour at all. Detection was unchanged after the fix — union recall stayed at 1.000 across five attack families.

So every detector in agentnorm answers "what do I do about an entity I have not observed?" explicitly:

• pool where the quantity is comparable across agents (volume, sequence, novel_tool) — shrink toward the population rather than treating the newcomer as alien
• assert where no history is needed (scope) — entitlement is checked, not learned
• suppress where pooling would be wrong (rate) — run length is not comparable between a triage agent making three calls and a reporting agent making forty, so agentnorm reports not yet calibrated instead of guessing

Thresholds you can act on

Thresholds come from a false-positive budget, not from maximising a statistic. An operator can act on "this fires once per two hundred clean runs". Nobody can act on "this maximises F1".

The budget is stated for the suite and divided across detectors, because five detectors each firing on 1% of runs union to about 5% — a suite advertised at 1% that delivers 5% gets muted within a week, and a muted detector detects nothing.

agentnorm also refuses to pretend about calibration. Asked for a 0.2% quantile from 40 runs, it fits, and warns:

calibration set has 40 runs but a 0.0020 quantile needs at least 500;
thresholds fall back to the observed maximum and the true false-positive
rate will exceed the budget

Design choices worth knowing

The unit is a run, not a span. Scope escalation, abnormal paths and volume anomalies are properties of a whole episode.

version is a change-point, not a label. Changing a prompt, model or tool grant changes the behavioural distribution. Baselines key on agent@version, so a deployment resets the baseline instead of triggering an alert storm.

Human and agent traces are never pooled. Their tool-use distributions are nothing alike, and pooling would poison the priors. actor_kind exists for this.

Failed calls are recorded, then re-raised. A trace that omits failures hides exactly the behaviour worth detecting — an agent probing for a tool it lacks, or retrying a refused action.

Attribute names follow OpenTelemetry's GenAI semantic conventions where they exist, so traces can be exported rather than trapped.

Measuring your own detectors

agentnorm.evaluation scores a suite against your labelled runs and sweeps the settings that were chosen by judgement rather than derived:

from agentnorm.evaluation import sensitivity, format_report

print(format_report(sensitivity(benign_runs, labelled_attacks)))

Run against the reference deployment's 4,000 benign and 200 labelled anomalous runs, recall was 1.000 at every setting — across a 64-fold range of prior strength, a 10-fold range of false-positive budget, and three calibration splits. No hand-picked constant is load-bearing, and the false-positive rate tracks the budget while staying below it, so the per-detector budget division is conservative rather than optimistic.

That result cuts both ways, and the second half matters more. Recall that cannot be moved by any setting also means the generated attacks sit nowhere near the decision boundary: the test is easy, which is evidence for the ceiling caveat below rather than against it.

Three checks on the monitor itself

A monitor is a model too, and three questions about it are usually answered by assertion. Can it fire at the budget it was calibrated to? Can the rate it reports be watched continuously without the watching itself causing false alarms? Is the model it watches still the one that was validated? Each has a function here, standard library only, and a reproducible experiment under tools/ whose output is committed in tools/results/.

Can it fire? Detection power at the calibrated budget

A false positive budget says how often a monitor fires on clean runs. A threshold of infinity meets every budget, so the other half of the result is the true positive rate at that calibration. detection_power plants one anomaly into each held out benign run and counts what fires, per detector and per anomaly type, with exact (Clopper Pearson) intervals for the true positive rate and for the false positive rate on the same runs before planting.

from agentnorm import detection_power, measure_power
from agentnorm.power import Burst, OversizedResult

monitor, report = measure_power(benign_runs, budget=0.01)   # fit, calibrate, hold out
print(report.format())
print(report.blind_spots(target=0.8))    # injectors shown to stay below 80%

report = detection_power(monitor, held_out, [OversizedResult(10), Burst(3)])

Measured on 2,400 held out runs from a synthetic population (three agents, three tenants with opaque scopes, heavy tailed result sizes; 8,000 runs; tools/exp_detection_power.py), at a 1% budget, 95% exact intervals:

planted anomaly	any detector	by detector
none (false positive rate)	0.007 [0.004, 0.012]
3 extra copies of normal calls	0.987 [0.982, 0.991]	sequence 0.936, rate 0.758
a tool never seen	1.000 [0.998, 1.000]	novel_tool 1.000, sequence 1.000
a scope owned by another tenant	1.000 [0.998, 1.000]	scope 1.000
one result 100 times its size	0.693 [0.674, 0.711]	volume 0.690
the same call 10 times in a row	1.000 [0.998, 1.000]	rate 1.000, sequence 0.706

The magnitude sweep is where the honest numbers are. One result 10 times its usual size is caught 4.8% [4.0, 5.8] of the time at a 1% budget, and still only 30.7% [28.8, 32.6] at a 20% budget. Thirty times: 26.1% and 67.1%. Result sizes are heavy tailed, so the volume threshold sits 4.5 standard deviations out and anything short of a hundredfold jump in a single call mostly passes. Under cold start (an agent version never seen) sequence and rate are suppressed by design, and bursts and extra calls are caught in 0 of 1,000 runs [0, 0.004]: the price of not alerting on every deployment.

The population is synthetic and the anomalies are the ones someone thought of, so this is an upper bound against an adversary who adapts, not an estimate.

Watching a rate without inflating false alarms

An operator checks the share of flagged runs after every run. A fixed sample test re run at every look is not a test at level alpha: each look is another chance, and a healthy, stationary stream eventually crosses. RateAlarm tests "is the rate above its baseline?" with a mixture martingale (Robbins; Howard et al. 2021), so the probability of ever raising a false alarm is at most alpha however often it is checked. RateConfidenceSequence is the matching two sided interval, valid at every time at once.

from agentnorm import RateAlarm, RateConfidenceSequence

alarm = RateAlarm(baseline=0.01, alpha=0.05)     # null: flag rate at most 1%
cs = RateConfidenceSequence(alpha=0.05)
for run in stream:
    flagged = monitor.score(run).flagged          # or a wrong answer label
    cs.update(flagged)
    if alarm.update(flagged):
        page(f"flag rate above 1%: e value {alarm.e_value:.0f}, rate in {cs.interval()}")

alarm.threshold(400)     # flags among the first 400 runs that would raise the alarm

Probability of any false alarm within 10,000 runs of a stationary 1% stream, alpha 0.05, 20,000 simulated streams per row (95% exact interval), and the same probability computed exactly by dynamic programming (tools/exp_anytime_alarms.py):

rule	simulated	exact
exact binomial test after every run	0.349 [0.342, 0.356]	0.344
the same, starting after 100 runs	0.298 [0.292, 0.304]	0.295
normal approximation z test after 30 runs	0.442 [0.435, 0.449]	0.436
one test, once, at run 10,000	0.043 [0.040, 0.046]	0.042
confidence sequence lower bound above 1%	0.011 [0.010, 0.013]	0.011
RateAlarm	0.031 [0.028, 0.033]	0.031

At a 5% base rate the rows are 0.402, 0.343, 0.454, 0.048, 0.013 and 0.035. The cost is detection delay, and it grows with how long the stream was clean before the change, because clean history is evidence too. For a rate doubling from 1% to 2%, median runs from the change to the alarm were 879, 1,460 and 2,724 when the change came at run 100, 1,000 and 5,000 (the naive test, which is not valid: 350, 781 and 1,553). RateAlarm was faster than the confidence sequence in every shift tested, as a one sided test should be.

Did the model behind the API change?

A hosted model can change without its name changing, and under a model risk framework such as OSFI E-23 a model change is a revalidation trigger. Following Gao, Liang and Guestrin (ICLR 2025, model equality testing), keep a fixed set of canary prompts, record the outputs once, collect them again later and run a two sample test. model_equality_test is a kernel (MMD) permutation test on style features of the output text (character trigrams and length). The default paired design swaps the two outputs within each prompt, which is exact under "same model" for any number of prompts.

from agentnorm import model_equality_test
from agentnorm.equality import pair_outputs

ids, before, after = pair_outputs(reference_outputs, todays_outputs)   # {prompt_id: text}
result = model_equality_test(before, after)
if result.rejected(0.01):
    open_revalidation_ticket(str(result))

Measured on real outputs recorded by the Plimsoll harness (same questions, same corpus, temperature 0; no API calls here; tools/exp_model_equality.py). Rejection rate at 0.05 over 500 random canary subsets of a pool of about 100 prompts, paired design, 95% exact interval:

comparison	5 prompts	10	20	50
Sonnet 4.6 vs Nova Pro, same retriever	0	0.994 [0.983, 0.999]	1.000	1.000
Haiku 4.5 vs Nova Lite, same retriever	0	0.466 [0.422, 0.511]	0.960 [0.939, 0.975]	1.000
Nova Lite vs Nova Pro (retriever differs too)	0	0.244 [0.207, 0.284]	0.590 [0.546, 0.634]	1.000
Haiku 4.5, two retrieval pipelines	0	0.006	0.020	0.014
one model, random disjoint halves (exact null)	0	0.034 to 0.060	0.036 to 0.050	0.040 to 0.046

Five prompts reject nothing, and that is arithmetic rather than bad luck: with one output per prompt the paired test has 2^(n-1) relabellings, so its smallest p value at n = 5 is 1/16. EqualityResult.min_p_value says so. The unpaired design can reject at 5 (0.450 for Sonnet against Nova Pro) but is weaker from 10 prompts on. On the full pool every different model pair gave p = 0.001, the smallest attainable with 999 permutations.

What the null is, and is not. There are no repeated samples of one model on the same prompts, so "same model, same prompts" cannot be tested directly; at temperature 0 it would mostly give identical text. The random halves row is exact by construction and checks calibration, not realism. The two pipeline rows are the realistic check: same model, same questions, different retrieved context, and the test almost never fires (full pool p of 0.29 and 0.55), so the style features track the model rather than the content. The flip side: a pipeline change of that kind would go unnoticed. And a rejection says the outputs changed, not why; hold the rest of the pipeline fixed when collecting canaries.

Alternatives

This is a small library in a category that is filling up. Worth knowing before you pick it:

	what it is	pick it over agentnorm when
AgentOps	mature agent monitoring SDK: cost tracking, benchmarking, broad framework coverage	you want the widest framework support and a hosted product behind it
Agentomaly	runtime behavioural anomaly detection over OpenTelemetry, with Slack/PagerDuty/Jaeger wiring	you already run OTel and want alerting plumbed into existing infrastructure
AgentLens	MCP-native observability with an append-only hash-chained audit log	you want a platform rather than a library, and MCP is your primary surface
TRACE	hardware-attested trust records binding model, policy and tool calls into a signed artifact	you need offline third-party verification and can run in a TEE — this is strictly stronger than hash chaining alone

What agentnorm does differently. Two things, and they are narrow:

Cold start is handled explicitly. The others learn a baseline from historical traces — Agentomaly's trainer takes min_traces=100 — and do not say what happens to an agent with no history. Measured here: a suite without cold-start handling alerts on 100% of known-benign runs from an unseen agent version. Since agent versions change weekly, that is the normal operating condition rather than an edge case. agentnorm decides per detector whether a quantity may be pooled across identities, asserted without history, or must be suppressed as uncalibrated — and reports which.

Zero dependencies. Enforced in CI by walking the AST, not asserted in prose. It installs into a locked-down environment and adds nothing to your dependency tree.

Where it is weaker. Fewer integrations than AgentOps or Agentomaly. No hosted backend, no dashboard, no alert routing. And hash chaining gives integrity, not the offline third-party verification TRACE achieves with hardware attestation.

Status

Early, and honest about it. The detectors are validated against five labelled attack families and against real agent traffic from one deployment. What is not yet true:

• the attack families are generated, so recall against them is a ceiling rather than a performance claim — a genuinely novel attack will not look like any of the five
• real-traffic validation is one deployment and tens of runs, not thousands
• there is no persistence layer; bring your own store

Reference deployment

agentnorm was extracted from Sentinel, an agentic IT-operations platform running on a live multi-cloud estate. Sentinel is where these numbers come from, and where agentnorm found a real cross-tenant disclosure in Sentinel's own agent — a run scoped to one tenant returning another tenant's host and package data, because tenant was a parameter the model could set freely. A boundary the caller can rewrite is not a boundary.

Licence

MIT.