Skip to main content

🛡️ Agent Cover

PyPI version Tests codecov Docs License: MIT Code style: ruff

Agent Cover is an observability and testing framework designed for LLMOps. Unlike traditional tools that measure executed lines of code, Agent Cover measures the Logical Coverage of your Agent.

It verifies that your test suite actually exercises the agent's capabilities: its Prompts, its Tools, and its Business Decision branches.

📚 Read the Documentation


📉 The Gap in LLM Testing

Traditional tools (like coverage.py) measure Python execution. However, in LLM-based applications, "line coverage" creates a false sense of security.

The Scenario: Your agent has a tool refund_user and a prompt instruction to "Ask for confirmation".

The Problem: Your test runs the code, but the LLM decides to skip the tool or ignore the prompt. Standard coverage says 100%. Agent Cover says 0% Tool Coverage.

🚀 How It Works

Agent Cover operates on a hybrid "Static + Runtime" model:

  1. 🔍 Inventory Scan (Static): Before tests run, it scans your codebase to build a map of "Logical Assets":

    • Prompts: LangChain objects, Jinja2 templates, and raw global strings.
    • Tools: Functions decorated with @tool or inheriting from BaseTool.
    • Decisions: Automatically derives coverage rules from your Pydantic models (Enums, Literals, Booleans).
  2. ⚡ Instrumentation (Runtime): It hooks into the execution lifecycle (LangChain, LlamaIndex, OpenAI, PromptFlow) to verify if those assets were actually used by the LLM.


📦 Installation

pip install agent-cover

For development and testing support:

pip install "agent-cover[dev]"

⚡ Quick Start

Create an agent-cover.yaml to define your Business Logic requirements.

# agent-cover.yaml
decisions:
  - id: intent_classification
    description: "Ensure the agent triggers all support paths"
    target_field: intent
    expected_values: ["REFUND", "TECH_SUPPORT", "SALES"]

2. Run with Pytest

Simply add the --agent-cov flag to your existing test run.

pytest --agent-cov --agent-cov-html=coverage_report

3. View the Report

Open coverage_report/index.html. You will see:

  • Prompt Coverage: Which templates were formatted and sent to the LLM.
  • Tool Coverage: Which tools the LLM actually decided to call.
  • Decision Coverage: Which logical branches (e.g., REFUND vs SALES) were observed in the output.

🔌 Supported Frameworks

Agent Cover dynamically patches these libraries to intercept logical events.

Framework Status Features Tracked
LangChain ✅ Chains, Agents, Tools, Callbacks, Pydantic Parsers
LlamaIndex ✅ AgentRunner, FunctionTools, Prompts
OpenAI SDK ✅ ChatCompletions, Function Calling (Raw Strings)
PromptFlow ✅ Flow Nodes, Jinja Templates, Python Tools
Pydantic ✅ Auto-generation of decision rules from Models

🛠 Advanced Usage: Microsoft PromptFlow

PromptFlow executes nodes in separate worker processes, which breaks standard coverage tools. Agent Cover includes a dedicated CLI wrapper to handle multi-process aggregation.

Instead of running pf run, wrap your command:

# Injects instrumentation into all child worker processes
agent-cover run --source-dir ./src -- pf run create --flow ./my_flow --data data.jsonl

This will automatically aggregate coverage data from all parallel workers into a single report.


🤝 Contributing

Contributions are welcome! Please ensure all pull requests pass the CI quality gates.

  1. Clone the repository.
  2. Install development dependencies: pip install -e ".[dev]"
  3. Install pre-commit hooks: pre-commit install
  4. Run the test suite: pytest

📄 License

This project is licensed under the MIT License.

Metadata

Release files for agent-cover 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for agent-cover 0.1.1
File Size Uploaded Provenance Actions
agent_cover-0.1.1.tar.gz 78.3 kB With provenance Details for agent_cover-0.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for agent-cover 0.1.1
File Interpreter ABI Platform Actions
agent_cover-0.1.1-py3-none-any.whl Python 3 none any Details for agent_cover-0.1.1-py3-none-any.whl

Total release size: 180.5 kB

Release files / agent_cover-0.1.1.tar.gz

Download URL agent_cover-0.1.1.tar.gz
Size 78.3 kB
Tags Source
SHA-256 checksum
How to use checksums
4739585f471278b4a6ebc1585148f2b11db681083696c6d32b28b1ca505850df
BLAKE2b-256 checksum
How to use checksums
5c5b422ce6818e9fd0abe560a354657aeb2dec41453de44a3ee32ae8c6a44b67
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jan 19, 2026.

Transparency log

Release files / agent_cover-0.1.1-py3-none-any.whl

Download URL agent_cover-0.1.1-py3-none-any.whl
Size 102.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
37cdb22ff31bfed0ae54475f7a9b85289c5d6ac5b70a8e97b9fcab14ad2163f7
BLAKE2b-256 checksum
How to use checksums
748d7702adc3d6d464f9f5e36b2f334ebc2a21fb79a227233e3f800d6ac64f65
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jan 19, 2026.

Transparency log

Security for agent-cover 0.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for agent_cover-0.1.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jan 19, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.1 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page