agentassure
pytest-based QA for enterprise AI agents. Talk to your agent in a test, then check what it said, which tools it called, what it changed in your system of record and whether it respected user permissions. Tests run live against the agent, combine across repeated runs and personas, and end in a quality gate your CI can block on.
- Responses: exact facts (
response_contains) and LLM-judged meaning (response_satisfies). - Behaviour: routing, tool calls and arguments, multi-agent handoffs, human escalation.
- Side effects: before/after diffs of real records; anything the agent changed that the test didn't expect fails.
- Safety: persona permissions, data exposure, prompt-injection resistance.
- Quality metrics: answer relevance, task completion, rubrics, faithfulness and context metrics (scored by your own judge LLM).
- Reports: console, JSON, JUnit and HTML (with full transcripts), plus optional OpenTelemetry export.
Works with Salesforce Agentforce (install agentassure-agentforce), any JSON-over-HTTP agent, or a Python
function.
Install
pip install AgentAssure-ai # HTTP / Python agents
pip install AgentAssure-ai AgentAssure-ai-agentforce # Salesforce Agentforce
pip install "AgentAssure-ai[otel]" # + OpenTelemetry export
Requires Python 3.11+. The agentassure command and the pytest plugin are registered on install.
Quick start
agentassure init --adapter http --env dev # agentassure.yaml, env/dev.yaml, .env.example, tests/
agentassure doctor --env dev # checks config, credentials and agent connectivity
agentassure test --env dev # runs the tests (any pytest args work: -k, -n 4, -x, ...)
A first test
import agentassure as aa
@aa.test(agent="service_agent", personas=["west_rep", "east_rep"])
def test_escalation(agent: aa.Conversation, sf: aa.StateProvider, expect: aa.Expect):
case = sf.create("Case", Subject="Delayed order", Priority="Medium") # removed after the run
with aa.step("Ask the agent to escalate"), sf.watch() as diff:
reply = agent.ask("Escalate case {n}", n=case.CaseNumber)
with aa.step("Check the reply and what changed"):
expect(reply).routed_to(topic="Case_Management").tool_called_once("Escalate_Case")
expect(reply).response_satisfies("Confirms the case is now high priority")
expect(diff).record(case).changed_only(Priority="High")
expect(diff).no_unexpected_mutations()
- Fixtures:
agent(the conversation),expect(soft assertions: every failure is reported, not just the first),state/sf(create, query and diff records) androw(for CSV-driven tests). The type hints are optional; they turn on autocomplete and hover docs in your editor. personas=[...]runs the test once per test user. Addrepeat=Nto run it N times and judge the combined results, since agents are non-deterministic.aa.step(...)groups the transcript and checks in the HTML report.
What you can check
| Area | Examples |
|---|---|
| Response | response_contains, response_not_contains, response_satisfies, response_max_length |
| Routing and tools | routed_to, tool_called, tool_called_once, tool_not_called, tool_order, no_tool_errors |
| Multi-agent | handoff, max_handoffs, no_handoff_loop, escalated_to_human |
| Permissions and data | respects_permissions, values_not_exposed, fields_not_exposed, identity_propagated |
| Security | attack_blocked, with ready-made payloads in aa.attacks |
| Record changes | created, created_only, record(...).changed_only, no_unexpected_mutations |
| LLM metrics | metric(aa.metrics.ANSWER_RELEVANCE), TASK_COMPLETION, RUBRIC, faithfulness, ... |
| Custom | expect.that(condition, "name"), expect.score(...), your own Metric subclass |
Data-driven tests
Tests are always Python. To drive one test from a spreadsheet, use
@aa.cases("cases.csv", unique_id="TestCaseId"), which runs the test once per row. Add
sequence_by="Turn" to group rows into multi-turn conversations.
Configuration
agentassure.yaml defines the agents and run-wide settings. env/<name>.yaml holds what differs per
environment: connection, personas (test users) and gates. ${VAR} and ${VAR:-default} read from the
environment or a .env file.
# agentassure.yaml
project: support-agents
agents:
order_bot:
adapter: http
base_url: ${ORDER_BOT_URL:-http://localhost:8080}
path: /chat
response: { text: reply, tool_calls: trace.tools }
judge: { provider: openai, model: gpt-4o-mini, api_key: "${AA_JUDGE_API_KEY}" }
metrics:
answer_relevance: 0.8
# env/qa.yaml
gates:
pass_rate: 0.95
critical_failures: 0
unexpected_mutations: 0
Command line
| Command | Does |
|---|---|
agentassure init |
scaffold a project (--adapter agentforce | http) |
agentassure doctor --env qa |
check configuration, credentials, org safety and agent resolution |
agentassure test --env qa |
run tests: --agent, --tags "smoke and not slow", --repeat N, plus any pytest args |
agentassure report results.json |
re-render the HTML report |
agentassure test exits 0 when the gates pass, 1 when a gate blocks or a test fails, 4 on invalid
configuration and 10 when the target is a production org.
Extending
Adapters, state providers and metrics are plugins, found through the agentassure.adapters,
agentassure.state_providers and agentassure.metrics entry-point groups. Subclass AgentAdapter,
StateProvider or Metric; each class's docstring lists what to implement. For a one-off metric, decorate
the subclass with @register_metric in a conftest.py.
Documentation
The full user guide (configuration, every check, CI setup, troubleshooting) is in docs/gitbook in the
source repository.
Metadata
Release files for AgentAssure-ai 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| agentassure_ai-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for agentassure_ai-0.1.0-py3-none-any.whl |
Release files / agentassure_ai-0.1.0-py3-none-any.whl
| Download URL | agentassure_ai-0.1.0-py3-none-any.whl |
|---|---|
| Size | 222.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
202df282ae05bdb8c0c68202e5b75522fcd4fcd8decd3b657f2432bb517b42f8
|
|
BLAKE2b-256 checksum How to use checksums |
422d0fec32e995b058a4f38012a06c6f4ec2dc7e362f348a578b32279bdab0f0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.32 {"installer":{"name":"uv","version":"0.11.32","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for AgentAssure-ai 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.