AgentICU — Multi-Agent Observability & Auto-Recovery

Instrument multi-agent LLM systems with zero code changes. AgentICU captures structured traces via OpenTelemetry, classifies failures in real-time using the MAST taxonomy (14 failure modes across 3 categories), runs automated recovery playbooks, and escalates when recovery fails.

Based on the MAST paper (NeurIPS 2025 Spotlight) — analysis of 1,642 agent traces across 7 frameworks found a 41–86.7% failure rate, with 79% of failures caused by specification ambiguity (44.2%) and coordination breakdowns (32.3%).

Quickstart

pip install "agenticu[tui]"

or, if you prefer npm (requires Python >= 3.11 on PATH):

npm install -g agenticu

Decorate any agent function — the @trace decorator emits OpenTelemetry spans automatically:

from agenticu import trace, tool_call

@trace(agent_id="researcher")
async def research_topic(query: str) -> str:
    results = await search_web(query)
    return await summarize(results)

@tool_call(return_snapshot=True)
async def search_web(query: str) -> str:
    """Simulate a search tool."""
    return f"Results for: {query}"

Enable inline auto-recovery by passing a classifier and store:

from agenticu import trace
from agenticu.classification.classifier import Classifier
from agenticu.collector.storage import TraceStore

@trace(agent_id="researcher", classifier=Classifier(fast_path_only=True), store=TraceStore())
async def robust_agent(query: str) -> str:
    # If this raises, AgentICU classifies the error,
    # runs the matching recovery playbook, and retries
    return await query_database(query)

Features

Architecture

Existing traces (Langfuse / Phoenix / raw OTel) — includes @decision rationale
               │
               ▼
    AgentICU Adapter Layer (OTel exporter)
               │
               ├──────────────► PROMAS-style Forecaster (predicts failure probability ahead of time)
               │
               ▼
    MAST Classifier (pattern-matching + embedding similarity, both active)
               │
               ▼
    Causal Attribution Layer (traces failure back through handoff chain to root-cause agent)
               │
               ▼
    Recovery Playbook Engine (targets the actual root-cause agent) ──► Escalation
               │
               ▼
    Outcome Tracking — feeds back into both the forecaster and the classifier

PROMAS Forecaster

Learns transition dynamics from your own collected traces and predicts the probability of reaching an error state before a handoff completes:

from agenticu import Forecaster, TransitionModel
from agenticu.collector.storage import TraceStore

store = TraceStore()
# ... collect traces ...

model = TransitionModel()
await model.learn(store)            # rebuild transition matrix from history
forecaster = Forecaster(model, threshold=0.3)

result = await forecaster.forecast_at_handoff("researcher", "writer")
print(result.failure_probability)   # 0.0–1.0
print(result.predicted_mode)        # most likely MAST mode
print(result.threshold_exceeded)    # trigger early intervention

Causal Attribution

When a failure is detected, trace it back through the handoff/decision chain to the agent that actually caused it — critical for pointing the recovery playbook at the right target:

from agenticu import CausalAttributor
from agenticu.collector.storage import TraceStore

store = TraceStore()
attributor = CausalAttributor(store)

result = await attributor.attribute(trace_id, failure_span_id)
print(result.root_agent_id)         # "agent_b" — the true root cause
print(result.attribution_chain)     # ["agent_a", "agent_b", "agent_c"]
print(result.confidence)            # 0.0–1.0

MAST Failure Taxonomy

CLI

# Start the API server
agenticu server

# Apply PostgreSQL schema
agenticu migrate --dsn postgresql://user:pass@localhost:5432/agenticu

# Health check
agenticu check

# Launch the terminal UI ops console (dashboard, traces, failures,
# forecast, attribution, recovery) — talks to the server above
agenticu tui
agenticu tui --server http://prod-host --api-key <key>

TUI

agenticu tui opens a full ops console in your terminal (requires the tui extra: pip install "agenticu[tui]", included in the npm install):

• Dashboard — server health, recovery success rate, OutcomeTracker status (auto-refreshes)
• Traces — filterable span table + trace detail with per-handoff forecasts
• Failures — filter by MAST mode / agent
• Forecast — predict P(failure) for any source → target handoff
• Attribution — trace a failure back to its root-cause agent
• Recover — trigger a recovery playbook for a trace + failure mode

AWS Deploy (OpenTofu)

cd infra/tofu
tofu init && tofu plan -var="container_image=<ECR>:latest" -out=tfplan
tofu apply tfplan

Provisions ECS Fargate (API + Grafana), RDS Postgres, ALB, ECR, Secrets Manager entries, and CloudWatch log groups. Full steps (state bucket, image push, DB migrate, TUI against prod, teardown, cost notes) are in infra/tofu/README.md.

Docker

docker compose up
# Starts: agenticu on :8000, PostgreSQL, Prometheus on :9090, Grafana on :3000
# The Grafana datasource + AgentICU dashboard are auto-provisioned.

API

Configuration

See config.yaml. Override via environment variables (AGENTICU__{SECTION}__{KEY}).

Development

git clone https://github.com/Anandhasasidharan/AGENTICU
cd agenticu
pip install -e ".[dev,all]"
pytest tests/ -v

License

MIT