Skip to main content

agentassure

pytest-based QA for enterprise AI agents. Talk to your agent in a test, then check what it said, which tools it called, what it changed in your system of record and whether it respected user permissions. Tests run live against the agent, combine across repeated runs and personas, and end in a quality gate your CI can block on.

  • Responses: exact facts (response_contains) and LLM-judged meaning (response_satisfies).
  • Behaviour: routing, tool calls and arguments, multi-agent handoffs, human escalation.
  • Side effects: before/after diffs of real records; anything the agent changed that the test didn't expect fails.
  • Safety: persona permissions, data exposure, prompt-injection resistance.
  • Quality metrics: answer relevance, task completion, rubrics, faithfulness and context metrics (scored by your own judge LLM).
  • Reports: console, JSON, JUnit and HTML (with full transcripts), plus optional OpenTelemetry export.

Works with Salesforce Agentforce (install agentassure-agentforce), any JSON-over-HTTP agent, or a Python function.

Install

pip install AgentAssure-ai                            # HTTP / Python agents
pip install AgentAssure-ai AgentAssure-ai-agentforce  # Salesforce Agentforce
pip install "AgentAssure-ai[otel]"                    # + OpenTelemetry export

Requires Python 3.11+. The agentassure command and the pytest plugin are registered on install.

Quick start

agentassure init --adapter http --env dev   # agentassure.yaml, env/dev.yaml, .env.example, tests/
agentassure doctor --env dev                # checks config, credentials and agent connectivity
agentassure test --env dev                  # runs the tests (any pytest args work: -k, -n 4, -x, ...)

A first test

import agentassure as aa


@aa.test(agent="service_agent", personas=["west_rep", "east_rep"])
def test_escalation(agent: aa.Conversation, sf: aa.StateProvider, expect: aa.Expect):
    case = sf.create("Case", Subject="Delayed order", Priority="Medium")  # removed after the run

    with aa.step("Ask the agent to escalate"), sf.watch() as diff:
        reply = agent.ask("Escalate case {n}", n=case.CaseNumber)

    with aa.step("Check the reply and what changed"):
        expect(reply).routed_to(topic="Case_Management").tool_called_once("Escalate_Case")
        expect(reply).response_satisfies("Confirms the case is now high priority")
        expect(diff).record(case).changed_only(Priority="High")
        expect(diff).no_unexpected_mutations()
  • Fixtures: agent (the conversation), expect (soft assertions: every failure is reported, not just the first), state / sf (create, query and diff records) and row (for CSV-driven tests). The type hints are optional; they turn on autocomplete and hover docs in your editor.
  • personas=[...] runs the test once per test user. Add repeat=N to run it N times and judge the combined results, since agents are non-deterministic.
  • aa.step(...) groups the transcript and checks in the HTML report.

What you can check

Area Examples
Response response_contains, response_not_contains, response_satisfies, response_max_length
Routing and tools routed_to, tool_called, tool_called_once, tool_not_called, tool_order, no_tool_errors
Multi-agent handoff, max_handoffs, no_handoff_loop, escalated_to_human
Permissions and data respects_permissions, values_not_exposed, fields_not_exposed, identity_propagated
Security attack_blocked, with ready-made payloads in aa.attacks
Record changes created, created_only, record(...).changed_only, no_unexpected_mutations
LLM metrics metric(aa.metrics.ANSWER_RELEVANCE), TASK_COMPLETION, RUBRIC, faithfulness, ...
Custom expect.that(condition, "name"), expect.score(...), your own Metric subclass

Data-driven tests

Tests are always Python. To drive one test from a spreadsheet, use @aa.cases("cases.csv", unique_id="TestCaseId"), which runs the test once per row. Add sequence_by="Turn" to group rows into multi-turn conversations.

Configuration

agentassure.yaml defines the agents and run-wide settings. env/<name>.yaml holds what differs per environment: connection, personas (test users) and gates. ${VAR} and ${VAR:-default} read from the environment or a .env file.

# agentassure.yaml
project: support-agents
agents:
  order_bot:
    adapter: http
    base_url: ${ORDER_BOT_URL:-http://localhost:8080}
    path: /chat
    response: { text: reply, tool_calls: trace.tools }
judge: { provider: openai, model: gpt-4o-mini, api_key: "${AA_JUDGE_API_KEY}" }
metrics:
  answer_relevance: 0.8
# env/qa.yaml
gates:
  pass_rate: 0.95
  critical_failures: 0
  unexpected_mutations: 0

Command line

Command Does
agentassure init scaffold a project (--adapter agentforce | http)
agentassure doctor --env qa check configuration, credentials, org safety and agent resolution
agentassure test --env qa run tests: --agent, --tags "smoke and not slow", --repeat N, plus any pytest args
agentassure report results.json re-render the HTML report

agentassure test exits 0 when the gates pass, 1 when a gate blocks or a test fails, 4 on invalid configuration and 10 when the target is a production org.

Extending

Adapters, state providers and metrics are plugins, found through the agentassure.adapters, agentassure.state_providers and agentassure.metrics entry-point groups. Subclass AgentAdapter, StateProvider or Metric; each class's docstring lists what to implement. For a one-off metric, decorate the subclass with @register_metric in a conftest.py.

Documentation

The full user guide (configuration, every check, CI setup, troubleshooting) is in docs/gitbook in the source repository.

Metadata

Release files for AgentAssure-ai 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distribution (wheel)

Table of built distributions (wheels) for AgentAssure-ai 0.1.0
File Interpreter ABI Platform Actions
agentassure_ai-0.1.0-py3-none-any.whl Python 3 none any Details for agentassure_ai-0.1.0-py3-none-any.whl

Release files / agentassure_ai-0.1.0-py3-none-any.whl

Download URL agentassure_ai-0.1.0-py3-none-any.whl
Size 222.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
202df282ae05bdb8c0c68202e5b75522fcd4fcd8decd3b657f2432bb517b42f8
BLAKE2b-256 checksum
How to use checksums
422d0fec32e995b058a4f38012a06c6f4ec2dc7e362f348a578b32279bdab0f0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.11.32 {"installer":{"name":"uv","version":"0.11.32","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for AgentAssure-ai 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

1 release file No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page