Agent Diagnostician

When an LLM agent fails, the run log usually tells you that it failed — not why.

Agent Diagnostician reads a JSON trace (task, steps, tool calls, outputs) and returns a diagnosis: failure type, subtype, confidence, evidence, and a suggested fix direction. It does not depend on LangChain, LangGraph, or any other agent framework.

Python 3.11+.

from agent_diagnostician import Classifier
from agent_diagnostician.tracer import load_fixture
from agent_diagnostician.reporter import Reporter

classifier = Classifier()  # no API key; uses rules + embeddings
trace = load_fixture("test cases/context_loss/context_loss__grounding__dropped_value.json")
Reporter.print(classifier.diagnose(trace))

Pass an LLM judge when you want the third detection tier (ambiguous tool use, goal misinterpretation, hallucination, and similar cases). Without a key, those paths stay conservative rather than calling an API.

Install

pip install agent-diagnostician

From a clone of this repo:

pip install -e .
pip install -e ".[llm-openai]"      # optional
pip install -e ".[llm-anthropic]"   # optional

The first diagnose() download of the embedding model (all-MiniLM-L6-v2 via Hugging Face) is about 90MB and needs network access. After that, embeddings run locally.

Core install currently includes Pydantic, sentence-transformers, scikit-learn, and the Gemini SDK (default live provider). OpenAI and Anthropic are extras.

Quick start

Build a trace in code:

from agent_diagnostician import Classifier
from agent_diagnostician.models.trace import AgentTrace, Step
from agent_diagnostician.reporter import Reporter

trace = AgentTrace(
    run_id="run_001",
    task="Refund order ORD-123 for $49.99",
    status="failed",
    total_steps=1,
    final_output=None,
    steps=[
        Step(
            step_index=0,
            tool_name="issue_refund",
            tool_input={"order_id": "ORD-999", "amount": 49.99},
            tool_output={"error": "Order not found"},
        ),
    ],
)

classifier = Classifier()
result = classifier.diagnose(trace)
Reporter.print(result)
print(result.failure_type.value, result.subtype, result.confidence_score)

Or load one of the synthetic fixtures in test cases/:

from agent_diagnostician.tracer import load_fixture

trace = load_fixture("test cases/context_loss/context_loss__grounding__dropped_value.json")

More examples: examples/basic_usage.py.

Optional: live LLM judge

Ambiguous cases can call Gemini, OpenAI, or Anthropic. Classifier() never does this on its own.

export LLM_PROVIDER=gemini          # gemini | openai | anthropic | mock
export LLM_API_KEY=your-key         # or GEMINI_API_KEY / OPENAI_API_KEY / ANTHROPIC_API_KEY
export LLM_MODEL=gemini-3.5-flash-lite

from agent_diagnostician import Classifier
from agent_diagnostician.analysis.llm import create_llm_judge_from_env

classifier = Classifier(llm_judge=create_llm_judge_from_env())

Smoke test: python scripts/configure_llm.py --test. Rate limits and mock vs live behavior: docs/TROUBLESHOOTING.md.

Run a subset of detectors:

from agent_diagnostician.models.enums import FailureType

classifier = Classifier(
    enabled_detectors=[FailureType.TOKEN_EXHAUSTION, FailureType.INFINITE_LOOP],
)

What it detects

Failure	Typical question it answers
Tool use	Wrong tool, bad parameters, or wrong values?
Goal satisfaction	Did the agent ignore a constraint or misread the task?
Hallucination	Did a step invent facts not in the tools or context?
Context loss	Did later steps drop earlier information?
Token exhaustion	Did the run die on context-length / token errors?
Premature termination	Did it stop before the task was done?
Infinite loop	Same tool, same error, or spinning reasoning?

The classifier runs the enabled detectors and returns the single strongest diagnosis (with a fixed priority order on ties).

Most detectors try cheap signals first, then embeddings, then an LLM only if needed:

1. Rules — schemas, constraints, error text
2. Embeddings — similarity (tool ranking, thought vs output, grounding)
3. LLM judge — structured prompts when the first two are inconclusive

Token exhaustion and infinite loop stay on rules/embeddings (no LLM). Hallucination uses the judge on each step. Thresholds live in agent_diagnostician/config.py. Pipeline detail: docs/ARCHITECTURE.md.

Evaluation

53 / 60 (88.3%) on synthetic JSON traces in test cases/ (Gemini gemini-3.5-flash-lite where a live judge was required). 36 fixtures resolved without an API call; 24 used live LLM.

Category	Fixtures	Passed
Context loss	6	6 (100%)
Infinite loop	9	9 (100%)
Hallucination	9	9 (100%, live judge)
Premature termination	6	6 (100%)
Classifier (multi-detector)	5	4 (80%)
Goal satisfaction	11	9 (81.8%)
Tool use	14	10 (71.4%)

These traces are synthetic and included in the repo so you can re-run the numbers. Known misses (7 fixtures): goal-failure aggregator edges, one classifier tiebreaker, and four tool-use fallback / negative-control cases. Per-fixture detail: docs/evaluation/SUMMARY.md.

python scripts/run_phased_fixture_evaluation.py \
  --initial-used 0 \
  --first-batch 3 \
  --batch-size 15 \
  --output docs/evaluation

Methodology: docs/evaluation/README.md.

Trace format

Minimum JSON:

{
  "run_id": "run_001",
  "task": "User task text",
  "status": "failed",
  "total_steps": 2,
  "final_output": null,
  "steps": [
    {
      "step_index": 0,
      "tool_name": "search",
      "tool_input": {"query": "..."},
      "tool_output": {"results": []},
      "thought": "optional reasoning",
      "error_message": "optional"
    }
  ]
}

Optional: available_tools, constraints, constraint_list, token counts. Schema: agent_diagnostician/models/README.md. Adapters: docs/INTEGRATIONS.md.

Docs

Document	Contents
API	Classes, methods, enums
Architecture	Pipeline and detection tiers
Integrations	LangChain and other frameworks
Performance	Embeddings, batching, cost
Troubleshooting	API keys, quotas, mock vs live
Evaluation	How the fixture benchmark is run
Contributing	Development workflow

License

MIT — see LICENSE.