agenticlens 0.1.2

Profile, analyze, and optimize token consumption in LLM-powered applications and agentic workflows.

AgenticLens

AgenticLens is an open-source Python profiler for LLM applications and agentic workflows. It helps developers understand where tokens, latency, and cost are spent, then turns that profile into actionable budget optimization recommendations.

Think of it as a lightweight, local cProfile for AI workflows: no hosted dashboard, no required backend, no account, and no data egress just to inspect a run.

Why AgenticLens?

LLM applications rarely spend money in one place. Cost often leaks across planners, retrievers, memory, tool calls, repeated system prompts, and final response steps.

Most observability tools can show token usage. AgenticLens focuses on the next question:

What should I change to reduce the bill?

AgenticLens currently detects patterns such as:

• repeated system prompts that may be cached or deduplicated
• excessive retrieved chunks in RAG workflows
• low-utility retrieved chunks that appear unlikely to affect the final answer
• long conversation history that should be summarized or truncated
• duplicate tool calls that should be cached
• projected token, dollar-per-run, and monthly savings

Status

AgenticLens is early-stage software. The core profiling, cost calculation, export, CLI, and rule-based recommendation engine are implemented, but the API may still evolve before a stable 1.0 release.

Installation

For local development from this repository:

git clone https://github.com/agenticlens/agenticlens.git
cd agenticlens
uv sync --extra dev

If you do not use uv, install in editable mode with development extras:

python -m venv .venv
. .venv/bin/activate
pip install -e ".[dev]"

On Windows PowerShell:

python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -e ".[dev]"

Quickstart

Instrument your workflow with explicit profile() and step() blocks:

from agenticlens import profile, step

with profile("Customer Support Agent"):
    with step(
        "Planner",
        type="planner",
        provider="openai",
        model="gpt-4o-mini",
        prompt=planner_prompt,
    ) as s:
        response = planner_llm.invoke(planner_prompt)
        s.record(response)

    with step(
        "Retriever",
        type="retriever",
        chunk_count=12,
        avg_tokens_per_chunk=80,
    ):
        chunks = retriever.search(user_question)

    with step(
        "Final Answer",
        type="final_response",
        provider="openai",
        model="gpt-4o-mini",
        final_answer="Refunds are processed to the original payment method.",
    ) as s:
        response = answer_llm.invoke(final_prompt)
        s.record(response)

Then profile and analyze a script:

uv run agenticlens profile examples/recommendations_demo.py --save workflow.json
uv run agenticlens analyze workflow.json

Example output:

Budget Optimization Run cost: $0.0068; reducible: ~$0.0024/run (35%), ~$2.38/month.

Optimization Suggestions
  * Long conversation history
  * Excessive retrieved chunks
  * Repeated system prompt
  * Low-utility retrieved chunks
  * Duplicate tool call

Estimated Savings: 35%

Core Concepts

Workflow

A workflow is one complete execution of an LLM application, such as answering a customer support question or running a multi-agent task.

with profile("Refund Support"):
    ...

Step

A step is a meaningful unit inside that workflow: planner, retriever, memory, tool call, LLM call, or final response.

with step("Retrieve Policy Chunks", type="retriever", chunk_count=10):
    ...

Recommendation

A recommendation is a rule-based optimization suggestion. Recommendations carry token savings, estimated percentage savings, dollar impact when pricing is known, confidence when relevant, and quality-risk notes for heuristics such as RAG chunk utility.

Features

Area	Capability
Profiling	Explicit profile() and step() context managers
Metrics	Prompt tokens, completion tokens, total tokens, latency, TPS, cost
Providers	OpenAI and Anthropic response usage extraction
Costing	Local pricing table plus user pricing overrides
Recommendations	Repeated prompts, excessive chunks, low-utility chunks, long history, duplicate tool calls
Budget impact	Dollar-per-run and monthly savings projections
CLI	profile, report, and analyze commands
Export	JSON and CSV workflow export
Tooling	pytest, Ruff, mypy, GitHub Actions

Cost Calculation

AgenticLens does not pull live prices from the internet. It uses a local pricing table in src/agenticlens/config/pricing.yaml and this formula:

input_cost = (prompt_tokens / 1000) * input_price_per_1k
output_cost = (completion_tokens / 1000) * output_price_per_1k
total_cost = input_cost + output_cost

Pricing resolution order:

1. User-supplied pricing override
2. Bundled pricing.yaml
3. Unknown model -> cost is reported as None, not $0.00

This keeps reporting honest when model pricing is missing or stale.

RAG Chunk Utility

The RAG utility rule compares retrieved context with useful context. It can use explicit chunk metadata such as:

{"text": "...", "utility_score": 0.92}
{"text": "...", "used": True}
{"text": "...", "cited": False}

If explicit signals are unavailable, it falls back to lightweight word-overlap between retrieved chunks and the final answer. That fallback is intentionally conservative: it identifies potentially low-utility context, not chunks that are guaranteed safe to remove.

Examples

Run the recommendation demo:

uv run agenticlens profile examples/recommendations_demo.py --save workflow.json
uv run agenticlens analyze workflow.json

Other examples:

• examples/basic_usage.py
• examples/rag_customer_support_demo.py
• examples/multiagent_support_demo.py

Some examples call real provider APIs and require provider API keys.

CLI Reference

Profile a Python script:

uv run agenticlens profile app.py

Save a workflow report:

uv run agenticlens profile app.py --save workflow.json

Display a saved workflow:

uv run agenticlens report workflow.json

Analyze a saved workflow:

uv run agenticlens analyze workflow.json

Development

Install development dependencies:

uv sync --extra dev

Run the test suite:

uv run pytest

Run linting, formatting, and type checks:

uv run ruff check .
uv run ruff format .
uv run mypy

Useful targeted checks while working:

uv run ruff check src tests
uv run ruff format --check src tests

Project Structure

src/agenticlens/
  profiler/       workflow and step profiling
  metrics/        cost and performance calculation
  providers/      provider response usage extraction
  recommenders/   rule-based optimization suggestions
  exporters/      JSON and CSV exports
  cli/            Typer CLI and Rich rendering
  config/         pricing and settings
  models/         Pydantic data models

Roadmap

Near-term priorities:

• richer RAG utility scoring with citation, reranker, and embedding signals
• model-tier mismatch detection
• prompt caching opportunity detection
• integrations for LangChain, LangGraph, LiteLLM, and OpenAI Agents SDK
• OpenTelemetry and OpenInference trace import
• optional prompt compression handoff

See ROADMAP.md and AgenticLens_Spec.md for more detail.

Contributing

Contributions are welcome. Good first areas include:

• provider integrations
• recommender rules
• example workflows
• docs and tutorials
• export formats
• test coverage

Please read CONTRIBUTING.md before opening a pull request.

Security

Please report vulnerabilities privately. See SECURITY.md.

Code of Conduct

This project follows CODE_OF_CONDUCT.md.

License

AgenticLens is released under the MIT License. See LICENSE.