pisama 0.4.0

Multi-agent failure detection for production AI systems

Pisama

Find and fix failures in AI agent systems. No LLM calls required.

Pisama ships 32 core heuristic detectors plus framework-specific detectors for n8n, LangGraph, Dify, and OpenClaw, running locally with zero LLM cost on the heuristic tier. On the TRAIL benchmark, Pisama's purpose-built detectors reach 60.2% F1 (100% precision, zero false positives) vs. about 11% for general-purpose LLMs doing open-ended failure analysis. Recall is about 43%, so the heuristic tier alone misses over half of failures; the tiered pipeline escalates to LLM judges for better coverage on the rest.

Install

pip install pisama

Usage

from pisama import analyze

result = analyze("trace.json")  # also accepts dicts and JSON strings

for issue in result.issues:
    print(f"[{issue.type}] {issue.summary} (severity: {issue.severity})")
    print(f"  Fix: {issue.recommendation}")

CLI

pisama analyze trace.json          # Analyze a trace
pisama watch python my_agent.py    # Watch a live agent (pip install pisama[auto])
pisama replay <trace-id>           # Re-run detection on stored traces
pisama smoke-test --last 50        # Batch test recent traces
pisama detectors                   # List all 32 core detectors
pisama mcp-server                  # Start MCP server (pip install pisama[mcp])

MCP Server

Works in Cursor, Claude Desktop, Windsurf — no API key needed:

{
  "mcpServers": {
    "pisama": { "command": "pisama", "args": ["mcp-server"] }
  }
}

Detectors

32 core detectors plus framework-specific detectors for n8n, LangGraph, Dify, and OpenClaw. A representative selection:

Detector	What It Catches
loop	Infinite loops, retry storms, stuck patterns
coordination	Deadlocked handoffs, message storms
hallucination	Factual errors, fabricated tool results
injection	Prompt injection, jailbreak attempts
corruption	State corruption, type drift
persona_drift	Persona drift, role confusion
derailment	Task deviation, goal drift
context	Context neglect, ignored instructions
specification	Output vs. requirement mismatch
communication	Inter-agent message breakdown
decomposition	Poor task breakdown, circular dependencies
workflow	Unreachable nodes, missing error handling
completion	Premature completion, unfinished work
withholding	Suppressed findings, hidden errors
convergence	Metric plateau, regression, thrashing
overflow	Context window exhaustion
delegation	Delegation quality and task handoff failures
grounding	Claims not supported by source documents
retrieval_quality	Poor retrieval relevance or coverage
compaction_quality	Information loss during context compaction

Benchmark Results

TRAIL (trace-level failure detection, 148 traces):

Method	Joint Accuracy
GPT-5.4	11.9%
Gemini 3.1 Pro	6.8%
Pisama	60.2%

Who&When (ICML 2025, multi-agent attribution, 58 hand-crafted cases):

Method	Agent Accuracy	Step Accuracy
GPT-5.4 Mini	60.3%	22.4%
Pisama + Sonnet 4	60.3%	24.1%

Links

• Documentation
• GitHub
• Platform

License

MIT