AgentLens

Find the decision that broke your agent.
Local trace capture and evidence-based root-cause debugging.

Website · Quickstart · Supported providers · Contribute · Report a bug

Your agent picked the wrong tool, repeated a failed call, or lost its goal. AgentLens helps you follow the recorded evidence back to a specific step and a concrete change to test. When the trace cannot support a cause, it reports insufficient evidence rather than pretending certainty.

Beta support boundaries and current blockers are documented in the Modules 1-2 support contract and beta completion report.

• Capture: record supported model calls, tool selections/results, usage, and errors.
• Inspect: read the run in your terminal or open its local HTML timeline.
• Diagnose: get a category, original failed step, evidence, suggested fix, and confidence/source label.

Illustration of the deterministic offline demo, not a dashboard screenshot or customer run. See the text output or run agentlens demo --no-browser to reproduce it.

Local by default · Python 3.9+ · Open source / MIT · Beta

Install and Try

The 0.1.3 beta is available on PyPI. The Python package is named runlens; the import and CLI are agentlens. Start in a fresh virtual environment:

python -m venv .venv
source .venv/bin/activate
pip install runlens
agentlens doctor
agentlens demo --no-browser

On Windows, activate with .venv\Scripts\activate instead. Use the activated environment's CLI; a global agentlens command may belong to an older install.

The demo captures a simulated broken agent, saves its trace, and prints a diagnosis. It never calls a provider, even when API keys are set. No API key or account is needed for this demo. Doctor checks local mechanics, not real-world diagnosis quality.

Already installed an older version? Run python -m pip install --upgrade runlens. For source development instead, clone the repository and install with python -m pip install -e ".[openai,anthropic]".

Capture a Run

Two lines enable instrumentation; a decorator groups and saves the run. Calling init() alone does not persist a grouped run.

The base package includes the offline CLI, not a provider SDK. For the OpenAI example below, install the extra first and configure your own OPENAI_API_KEY outside the source file. Anthropic users can install runlens[anthropic] instead.

python -m pip install "runlens[openai]"

import agentlens
agentlens.init()

from openai import OpenAI
client = OpenAI()  # Uses your OPENAI_API_KEY for your own provider requests.

@agentlens.run(name="support_agent")
def support_agent(query):
    return client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": query}],
    )

support_agent("Find the status of customer C17.")

This is an instrumentation example, not a complete customer lookup agent. Your own OpenAI call requires OPENAI_API_KEY and provider access. Keep your existing model configuration. For provider-specific examples, see examples/.

agentlens.record_tool_result(
    tool_name="lookup", tool_use_id=call_id, input=arguments, output=result
)

Inspect and Diagnose

agentlens runs list
agentlens runs show <run_id>
agentlens diagnose <run_id>                    # Offline, even with API keys set
agentlens runs view <run_id>                   # Opens a local HTML timeline

agentlens runs prompt <run_id> --step 2        # Original trace step, not LLM-call ordinal
agentlens runs replay <run_id>
agentlens runs stitch <run_id>
agentlens watch                               # Changed saved snapshots, not live event streaming
agentlens stats                               # All runs; known cost subtotal and unknown-price count
agentlens similar <run_id>
agentlens clusters
agentlens anonymize <run_id>
agentlens feedback-template <run_id>
agentlens evaluate

Example from the deterministic demo, abbreviated:

SOURCE:
  Heuristic fallback
ROOT CAUSE:
  tool_selection
FAILED AT (ROOT-CAUSE STEP):
  Step 2 (search_web)
WHY:
  Step 2 called 'search_web', but the tool error explicitly identifies
  'query_db' as the required tool; no successful retry is recorded.
SUGGESTED FIX (not verified):
  Route this operation to 'query_db', distinguish the tool descriptions,
  and add a routing regression test.
EVIDENCE STRENGTH: observed
Confidence score (not a calibrated probability): 0.85

Categories: tool_selection, loop, cascade, context_pollution, state_drift, overflow. Local rules are intentionally narrow: explicit routing errors, repeated failed calls without progress, downstream reuse of flagged fields, or explicit evidence of conflicting instructions/goal/context loss. Names, topics, repetition alone, and preprocessing truncation are not proof. These rules do not reliably infer subtle semantic errors or model intent.

Scores express evidence strength, not calibrated odds of correctness. A completed process is not proof of a correct answer. Partial/running/cancelled runs retain their execution status. Absence of a detected failure is not a safety certification. Hallucination checks cover constrained schema violations and explicit same-entity field contradictions; they do not verify arbitrary facts.

Verified Capture Scope

Privacy and Remote Diagnosis

Local capture and default diagnosis store/process traces locally. Raw capture may contain secrets, personal information, tool output, and provider error text. Keep .agentlens/, .env, generated exports and private logs out of Git.

Remote diagnosis requires explicit opt-in:

agentlens diagnose <run_id> --provider openai
# or --provider anthropic

Programmatic equivalent: diagnose_run(run_json, provider="openai", timeout=10) from agentlens_engine.diagnose. The chosen provider receives a redacted compact trace containing selected prompts, tool definitions, inputs/outputs and errors. It receives no more than 60 KB; larger payloads fall back to local rules. Connection timeout: 3 seconds; read timeout: at most 10 seconds per request; SDK retries: zero; one extra attempt only for invalid diagnosis JSON. These are transport timeouts, not a hard total wall-clock deadline. Provider failures or invalid evidence fall back to local rules and are labeled. No provider is selected merely because a key exists.

Restricted remote contract: a model proposal must match the cause, original step, evidence and explanation/fix templates independently produced by the local structural rules. New claims, paraphrases, unrelated quotes and higher confidence are rejected with labeled heuristic fallback (or insufficient evidence). Accepted LLM proposals do not expand diagnosis coverage beyond those rules. Fixes remain unverified suggestions; confidence is not a calibrated probability. Live-provider and independent developer validation are still outstanding; see the beta report.

anonymize and upload prepare use the same best-effort credential/PII policy as server ingestion. Review exports manually: proprietary facts, arbitrary names, encoded secrets and unknown credential formats cannot be guaranteed removed. Preparation writes files locally; it does not upload them.

Optional person-name detection:

python -m pip install "runlens[pii]"
python -m spacy download en_core_web_sm

Without that model, built-in credential patterns and explicit PII keys still work, but free-text name detection is unavailable. The optional server remains an unauthenticated local/self-host component; do not expose it publicly. Its residual scanner rejects known patterns, not all possible sensitive information.

Evaluation and Beta Status

agentlens evaluate runs the packaged positive and healthy corpus offline, plus explicitly saved real_world_cases/ in the working directory. The packaged corpus has 6 synthetic positives and 8 healthy/abstention cases. The checkout also has 2 developer-operated OSS validation cases, not customer traces. The command reports raw category/step matches, false positives/negatives, abstentions and runtime. These small regression counts do not estimate accuracy on unseen users. No zero-false-positive or live-provider benchmark is claimed.

Evaluation now separates regression, internal_natural, external_developer and unclassified populations. Confident-wrong uses the existing 0.80 threshold with an explicit numerator/denominator; a wrong original step counts as wrong for that metric. Independent review instructions are in the case template.

What we are working on now: validating whether diagnoses help developers debug real broken agents faster. The next evidence we need is user-confirmed correctness, useful fixes, and repeat use, not more features or larger synthetic scores.

Real external users, confirmed usefulness and payment intent remain unverified. Regression success alone is not evidence of real-world accuracy or product-market fit. For case format and verification commands see release verification.

Help Test a Real Failure

Run AgentLens on a broken agent, then prepare feedback locally:

agentlens anonymize <run_id>
agentlens feedback-template <run_id>

Review the export manually before sharing anything. Tell us whether the failed step was right, the explanation made sense, the fix helped, and you would use it again. A sanitized description or synthetic reproduction is enough to start an issue; do not post raw traces or secrets.

Contributing

See contribution.md for setup, testing, pull request rules, privacy and security reporting, community expectations, and licensing.

License

AgentLens is available under the MIT License.