agentcheck
agentcheck: Trace ⋅ Replay ⋅ Test your AI agents like real software.
AgentCheck is a minimal but complete toolkit for tracing, replaying, diffing, and testing AI agent executions. Think of it as version control and testing for your AI agents.
🚀 Install
pip install agentcheck
⚡ Quickstart Demo
export OPENAI_API_KEY=sk-...
# 1️⃣ Capture baseline trace
python demo/demo_agent.py --output baseline.json
# 2️⃣ Modify the prompt inside demo_agent.py (e.g. change tone)
# 3️⃣ Replay with new code/model
agentcheck replay baseline.json --output new.json
# 4️⃣ See what changed
agentcheck diff baseline.json new.json
# 5️⃣ Assert the new output still mentions the user's name
agentcheck assert new.json --contains "John Doe"
Or run the complete demo:
cd demo && ./demo_run.sh
🎯 Features
| Feature | Description | CLI Command | Python API |
|---|---|---|---|
| Trace | Capture agent execution (prompts, outputs, costs, timing) | agentcheck trace <command> |
@agentcheck.trace() |
| Replay | Re-run trace against current code/model | agentcheck replay trace.json |
agentcheck.replay_trace() |
| Diff | Compare traces and highlight changes | agentcheck diff trace_a.json trace_b.json |
agentcheck.diff_traces() |
| Assert | Test trace contents (CI-friendly) | agentcheck assert trace.json --contains "foo" |
agentcheck.assert_trace() |
📖 Usage
Tracing with Decorator
import agentcheck
import openai
@agentcheck.trace(output="my_trace.json")
def my_agent(user_input: str) -> str:
response = openai.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": user_input}]
)
return response.choices[0].message.content
# Automatically traces execution and saves to my_trace.json
result = my_agent("Hello, world!")
Tracing with Context Manager
import agentcheck
with agentcheck.Trace(output="trace.json") as t:
# Your agent code here
messages = [{"role": "user", "content": "Hello"}]
# Manually add LLM calls to trace
response = openai.chat.completions.create(
model="gpt-4o-mini", messages=messages
)
t.add_llm_call(
messages=messages,
response={"content": response.choices[0].message.content, "usage": response.usage},
model="gpt-4o-mini"
)
CLI Commands
# Trace a Python script
agentcheck trace "python my_agent.py" --output trace.json
# Replay a trace with a different model
agentcheck replay trace.json --model gpt-4 --output new_trace.json
# Compare two traces
agentcheck diff baseline.json new_trace.json
# Assert trace contains expected content
agentcheck assert trace.json --contains "expected output"
# Assert with JSONPath
agentcheck assert trace.json --jsonpath "$.steps[-1].output.content" --contains "John"
# Assert cost and step constraints
agentcheck assert trace.json --max-cost 0.05 --min-steps 1 --max-steps 10
# Pretty-print a trace
agentcheck show trace.json
🏗️ Architecture
┌─────────────────┐ ┌──────────────┐ ┌─────────────────┐
│ Your Agent │───▶│ agentcheck │───▶│ trace.json │
│ │ │ tracer │ │ │
└─────────────────┘ └──────────────┘ └─────────────────┘
│
┌─────────────────────┼─────────────────────┐
▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ replay │ │ diff │ │ assert │
│ (re-execute) │ │ (compare) │ │ (test) │
└─────────────────┘ └─────────────────┘ └─────────────────┘
📋 Trace Format
AgentCheck uses a standardized JSON schema for traces:
{
"trace_id": "uuid",
"version": "1.0",
"start_time": "2024-01-01T12:00:00Z",
"end_time": "2024-01-01T12:00:05Z",
"metadata": {
"total_cost": 0.0023,
"function_name": "my_agent"
},
"steps": [
{
"step_id": "uuid",
"type": "llm_call",
"start_time": "2024-01-01T12:00:01Z",
"end_time": "2024-01-01T12:00:04Z",
"input": {
"messages": [...],
"model": "gpt-4o-mini"
},
"output": {
"content": "Agent response...",
"usage": {"prompt_tokens": 10, "completion_tokens": 20},
"cost": 0.0023
}
}
]
}
🧪 Testing & CI Integration
AgentCheck is designed for CI/CD pipelines:
# In your CI pipeline
agentcheck replay baseline_trace.json --output ci_trace.json
agentcheck assert ci_trace.json --contains "expected behavior" --max-cost 0.10
# Exit codes
# 0 = success
# 1 = assertion failed or error
🛠️ Development
# Install in development mode
git clone https://github.com/agentcheck/agentcheck
cd agentcheck
pip install -e ".[dev]"
# Run tests
pytest
# Format code
ruff format .
# Type check
mypy agentcheck/
📄 License
MIT License - see LICENSE file.
🤝 Contributing
Contributions welcome! Please see CONTRIBUTING.md for guidelines.
Built for the era of AI agents 🤖✨
Metadata
Release files for agentcheck 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| agentcheck-0.1.0.tar.gz | 18.8 kB | Without provenance | Details for agentcheck-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| agentcheck-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for agentcheck-0.1.0-py3-none-any.whl |
Total release size: 35.6 kB
Release files / agentcheck-0.1.0.tar.gz
| Download URL | agentcheck-0.1.0.tar.gz |
|---|---|
| Size | 18.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
99badbb62805315bff4a261aeb7888b9626652cc5792d66c132537405b9400be
|
|
BLAKE2b-256 checksum How to use checksums |
b77ca3e8acc3b7bdd5c733ece9e1c2b1dd376b97d5956623595e5d6927c02006
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.11.4
|
Release files / agentcheck-0.1.0-py3-none-any.whl
| Download URL | agentcheck-0.1.0-py3-none-any.whl |
|---|---|
| Size | 16.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
0b08b3c23c8eba4492703fe45fa216fd798df63857045579d5e8b64fb8a4af0d
|
|
BLAKE2b-256 checksum How to use checksums |
06735def14368c372fd8e086d8626bd4d70dd1a17b8611f576f55f86692c3110
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.11.4
|
Security for agentcheck 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.