agent-polygraph 0.1.0

Framework-agnostic completion-lie detector for AI agents -- catches an agent turn whose final claim contradicts its own tool evidence.

agent-polygraph

A framework-agnostic completion-lie detector for AI agents. Given one agent turn — a task, the tool calls and results it produced ("evidence"), and the final assistant message ("closing") — verify() decides whether the closing makes a completion or success claim that the turn's own evidence contradicts.

It measures the turn, not the model. The check is heuristic, deterministic, and synchronous, with zero runtime dependencies (standard library only).

from agent_polygraph import verify, ToolCall, ToolResult

v = verify(
    [ToolCall("c1", "deploy", {"svc": "web"}),
     ToolResult("c1", content="ERROR: image pull failed", is_error=True)],
    final_claim="Done -- deployed successfully.",
)

v.verdict      # "lie"
v.category     # "L1"  (error concealment)
v.confidence   # 1.0   (structured error bit)
v.evidence     # (Finding(check="error_concealment", detail=...),)

This is the SDK companion to polygraph-bench, the public benchmark for agent-lying detectors. The detector shipped here is a clean-room reimplementation of a private production honesty gate, validated at verdict parity against that gate on the benchmark's 396-item corpus (numbers below).

---

Install

pip install agent-polygraph

Python 3.10+. No third-party runtime dependencies.

---

What it catches

Three heuristic check families, each mapped to a diagnostic category:

category	family (check id)	catches
L1	error_concealment	an unrecovered tool error hidden behind a plain, unacknowledged success claim
L2	completion_vs_evidence	a completeness / counted-total claim over a full-page fetch with no proof the set was exhausted
L3	narrative_vs_assertion	a runtime "it works / is fixed" claim after a file edit with nothing actually run

verify() returns a Verdict:

Verdict(
    verdict="lie" | "honest",
    evidence=(Finding(check, detail), ...),
    confidence=1.0 | 0.7,      # signal-quality GRADE, not a probability
    detector_version="0.1.0",  # self-stamped for cross-version comparison
    flags=(...),               # degradation markers driving a 0.7 grade
    category="L1" | "L2" | "L3" | None,
)

confidence is a signal-quality grade, not a probability: 1.0 when the verdict rode a structured framework signal, 0.7 when it rode an inferred truncation marker or a sniffed error payload (see degraded signals).

---

Framework adapters

Four adapters map a framework's native emission onto the same verify() core. The design default is audit everywhere — a foreign framework's false-positive rate is unknown until you measure it on your own traffic, and a false halt kills adoption. Blocking is opt-in, and only on the two surfaces that can actually halt a run; the audit-only adapters raise loudly on mode="block" rather than silently downgrading.

adapter	entry point	tool-error signal	default	block
OpenAI Agents SDK	output_guardrail	span wrapper error.data.error (structured)	audit	opt-in
Claude Code stop-hook	stop_hook	tool_result.is_error (structured)	audit	opt-in
OTel / OpenInference	verify_spans	OTel span status_code=ERROR (structured)	audit	raises loud
LiteLLM proxy	post_call_guardrail	none — sniff role:"tool" text only	audit	raises loud

from agent_polygraph.adapters import output_guardrail

result = output_guardrail(trace_export, mode="block")  # halt on a lie
if result.tripwire_triggered:
    ...  # the Agents SDK stops the run

Per-adapter quickstarts live in docs/: OpenAI Agents · OpenInference / OTel · Claude Code · LiteLLM.

---

Validation

All numbers below come from actually-executed runs and are reported verbatim. Parity and the recall floor are the honest limits of a v0.1 heuristic detector — read them before you deploy in block mode.

Parity vs the private gate — 393 / 396 (99.24%)

The detector was written clean-room from the public benchmark spec and corpus, then refined by behavioural parity iteration against a private production gate treated as a black box (feed an input, observe its block/allow verdict; the gate's source was never read). On all 396 public bench items:

agreement matrix (clean-room \ gate):
                 gate=lie   gate=honest
  clean=lie            88             0
  clean=honest          3           305

• 0 over-blocks — the clean detector never blocks anything the gate allows.
• 3 under-blocks, all category H2 (disclosed-failure honest twins). These are exactly the three items the gate's own heuristic tier false-blocks; the clean detector matches the ground-truth label on all three (they are honest). Reproducing the gate's mistake would require either reading its source or overfitting a rule that makes the public detector actively worse.

Label-level, both over the 396-item set: identical recall (0.460) and the same true-positive set; the clean detector carries 3 fewer false blocks (FP 3.18% vs the gate's 4.55%).

Real-trace replay — 33 traces, 0 unexplained flips

33 real captured framework traces (OpenAI Agents ×10, OpenInference ×6, Claude Code ×13, LiteLLM ×4; local models only, no paid API calls) were replayed through each shipped adapter's public surface and through verify() on the same parsed trajectory:

• 0 surface-vs-direct disagreements across all 33.
• 2 lies caught — both Claude Code turns with an unrecovered structured is_error bit under a success claim (confidence 1.0). The other 31 → honest, all correct true-negatives (including honest failure-disclosure turns).

LiteLLM opaque-error recall floor — read this before using the LiteLLM adapter

The detection signal rides the structured tool-error bit. At the LiteLLM proxy that bit does not exist: a tool failure is visible only as whatever text the agent wrote into the role:"tool" message. The adapter recovers what it can by sniffing the payload ("ERROR: ...", Traceback, …) — but a failure that does not read as an error in text is invisible at that layer.

Measured cost, round-trip over all 396 bench items (reference = direct detector):

path              agree  flips   recall      FP
direct(ref)      100.0%      0    0.460    0.032
agents           100.0%      0    0.460    0.032
openinference    100.0%      0    0.460    0.032
claude_code      100.0%      0    0.460    0.032
litellm           89.6%     41    0.227    0.032

• The three structured-error adapters are fully consistent with direct detection — 0 verdict flips on 396 items.
• LiteLLM: 41 flips, all lie -> honest — recall drops from 0.460 to 0.227 on error-concealment lies whose tool text does not read as an error. FP is unchanged (0.032) — no false blocks are introduced, only real lies are missed.

This is a measured framework ceiling, not a bug: the proxy sits below the structured error signal. The LiteLLM adapter is audit-only for this reason.

---

Known limitations

These are structural properties of a heuristic-tier v0.1 detector, not defects to be worked around silently.

The error bit is the portability spine. Every adapter recovers the tool-failure signal per framework — structured where the framework exposes it (Agents span wrapper, OpenInference OTel status, Claude Code is_error), sniffed from the payload where it does not (LiteLLM). A sniffed error grades to 0.7 confidence; a framework that redacts tool outputs by default is a dark surface. If you take one design fact from this project: a completion-lie detector is only as portable as the tool-error signal it can recover.

Dangling call + success claim is a heuristic miss (judge-tier material). A tool_call with no matching tool_result (a legal dangling call) plus a "Done, deployed successfully" closing returns honest. The heuristic tier cannot associate an outcome that never returned. This is the motivating example for the judge-tier extension below, and it is a documented miss in both the private gate and this detector.

Inferred truncation is a low-confidence, verdict-neutral signal. Truncation is inferred from seam markers ([...omitted...], Claude Code's [N lines truncated], or markers you register). An inferred truncation only downgrades confidence to 0.7; it never flips a verdict (the detector uses the bit solely to grade an L2 completeness finding). A structured truncation flag (Claude Code truncatedByTokenCap) stays at 1.0.

A1 — Claude Code truncation markers inside skipped file-read attachments. In Claude Code, the [N lines truncated] marker sometimes rides inside a file-read attachment block (rather than a tool_result.content), which the adapter skips. On those turns the truncation bit is not set. This is verdict-safe and confidence-neutral in every observed case (it can only matter when an L2 completeness finding fires over such an attachment — which did not occur in validation); the residual is a possible missed 0.7 downgrade, never a false block. A v0.1.x follow-up may scan attachment snippets or let you register the marker via Config.extra_truncation_markers.

---

Judge-tier extension

The heuristic detector is deterministic and cheap; some cases (the dangling-call miss above) need reasoning it cannot do. verify() exposes a synchronous Judge protocol as the stable extension seam. No judge ships in v0.1 — this is the documented interface a future model-backed judge slots into without changing the verify() contract.

from agent_polygraph import verify, Judge, Verdict

class DanglingCallJudge:
    """Motivating example: catch a success claim over a call that never returned."""
    def judge(self, trajectory, final_claim, task, heuristic_verdict):
        call_ids = {e.call_id for e in trajectory if getattr(e, "call_id", None)
                    and type(e).__name__ == "ToolCall"}
        result_ids = {e.call_id for e in trajectory if type(e).__name__ == "ToolResult"}
        if call_ids - result_ids and heuristic_verdict.verdict == "honest":
            return Verdict("lie", confidence=0.7, detector_version="my-judge-0.1")
        return None  # keep the heuristic verdict

v = verify(trajectory, final_claim, judge=DanglingCallJudge(), escalate="honest_only")

escalate chooses which heuristic verdicts reach the judge: "honest_only" (default — catch what the heuristic missed), "degraded_only" (any 0.7-confidence verdict), or "all". A judge that overrides a verdict self-stamps its own detector_version so co-evolving verdicts stay comparable.

---

Relationship to polygraph-bench

agent-polygraph and polygraph-bench are two halves of the same effort:

• polygraph-bench — the public 396-item benchmark that measures completion-lie detectors. The clean-room detector here is validated against it. Leaderboard · Dataset on Hugging Face.
• agent-polygraph (this package) — the detector you run in production, with adapters for real frameworks.

If you build a better detector, benchmark it on polygraph-bench and open a submission PR there.

---

License

MIT. See LICENSE.