🛡️ Agent Cover
Agent Cover is an observability and testing framework designed for LLMOps. Unlike traditional tools that measure executed lines of code, Agent Cover measures the Logical Coverage of your Agent.
It verifies that your test suite actually exercises the agent's capabilities: its Prompts, its Tools, and its Business Decision branches.
📉 The Gap in LLM Testing
Traditional tools (like coverage.py) measure Python execution. However, in LLM-based applications, "line coverage" creates a false sense of security.
The Scenario: Your agent has a tool
refund_userand a prompt instruction to "Ask for confirmation".The Problem: Your test runs the code, but the LLM decides to skip the tool or ignore the prompt. Standard coverage says 100%. Agent Cover says 0% Tool Coverage.
🚀 How It Works
Agent Cover operates on a hybrid "Static + Runtime" model:
-
🔍 Inventory Scan (Static): Before tests run, it scans your codebase to build a map of "Logical Assets":
- Prompts: LangChain objects, Jinja2 templates, and raw global strings.
- Tools: Functions decorated with
@toolor inheriting fromBaseTool. - Decisions: Automatically derives coverage rules from your Pydantic models (Enums, Literals, Booleans).
-
⚡ Instrumentation (Runtime): It hooks into the execution lifecycle (LangChain, LlamaIndex, OpenAI, PromptFlow) to verify if those assets were actually used by the LLM.
📦 Installation
pip install agent-cover
For development and testing support:
pip install "agent-cover[dev]"
⚡ Quick Start
1. Configuration (Optional but Recommended)
Create an agent-cover.yaml to define your Business Logic requirements.
# agent-cover.yaml
decisions:
- id: intent_classification
description: "Ensure the agent triggers all support paths"
target_field: intent
expected_values: ["REFUND", "TECH_SUPPORT", "SALES"]
2. Run with Pytest
Simply add the --agent-cov flag to your existing test run.
pytest --agent-cov --agent-cov-html=coverage_report
3. View the Report
Open coverage_report/index.html. You will see:
- Prompt Coverage: Which templates were formatted and sent to the LLM.
- Tool Coverage: Which tools the LLM actually decided to call.
- Decision Coverage: Which logical branches (e.g.,
REFUNDvsSALES) were observed in the output.
🔌 Supported Frameworks
Agent Cover dynamically patches these libraries to intercept logical events.
| Framework | Status | Features Tracked |
|---|---|---|
| LangChain | ✅ | Chains, Agents, Tools, Callbacks, Pydantic Parsers |
| LlamaIndex | ✅ | AgentRunner, FunctionTools, Prompts |
| OpenAI SDK | ✅ | ChatCompletions, Function Calling (Raw Strings) |
| PromptFlow | ✅ | Flow Nodes, Jinja Templates, Python Tools |
| Pydantic | ✅ | Auto-generation of decision rules from Models |
🛠 Advanced Usage: Microsoft PromptFlow
PromptFlow executes nodes in separate worker processes, which breaks standard coverage tools. Agent Cover includes a dedicated CLI wrapper to handle multi-process aggregation.
Instead of running pf run, wrap your command:
# Injects instrumentation into all child worker processes
agent-cover run --source-dir ./src -- pf run create --flow ./my_flow --data data.jsonl
This will automatically aggregate coverage data from all parallel workers into a single report.
🤝 Contributing
Contributions are welcome! Please ensure all pull requests pass the CI quality gates.
- Clone the repository.
- Install development dependencies:
pip install -e ".[dev]" - Install pre-commit hooks:
pre-commit install - Run the test suite:
pytest
📄 License
This project is licensed under the MIT License.
Metadata
Release files for agent-cover 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| agent_cover-0.1.1.tar.gz | 78.3 kB | With provenance | Details for agent_cover-0.1.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| agent_cover-0.1.1-py3-none-any.whl | Python 3 | none | any | Details for agent_cover-0.1.1-py3-none-any.whl |
Total release size: 180.5 kB
Release files / agent_cover-0.1.1.tar.gz
| Download URL | agent_cover-0.1.1.tar.gz |
|---|---|
| Size | 78.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4739585f471278b4a6ebc1585148f2b11db681083696c6d32b28b1ca505850df
|
|
BLAKE2b-256 checksum How to use checksums |
5c5b422ce6818e9fd0abe560a354657aeb2dec41453de44a3ee32ae8c6a44b67
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jan 19, 2026.
Transparency logRelease files / agent_cover-0.1.1-py3-none-any.whl
| Download URL | agent_cover-0.1.1-py3-none-any.whl |
|---|---|
| Size | 102.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
37cdb22ff31bfed0ae54475f7a9b85289c5d6ac5b70a8e97b9fcab14ad2163f7
|
|
BLAKE2b-256 checksum How to use checksums |
748d7702adc3d6d464f9f5e36b2f334ebc2a21fb79a227233e3f800d6ac64f65
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jan 19, 2026.
Transparency logSecurity for agent-cover 0.1.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for agent_cover-0.1.1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jan 19, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.