agentqa-sdk

Regression testing and monitoring for AI agents — record every run, test every change, and catch failures before they break real workflows.

Install

pip install agentqa-sdk

(The package installs as agentqa-sdk; the import name is agentqa.)

Configure

export AGENTQA_API_KEY=aq_live_...        # from your project's Settings page
export AGENTQA_API_URL=https://your-agentqa-deployment

Record a run

import agentqa

with agentqa.track_run(
    agent_name="sales_research_agent",
    task_input="Research this lead and update the CRM",
) as run:
    run.log_tool_call("crm.lookup", {"company": "Acme"}, {"id": 42})
    run.log_llm_call("claude-sonnet-5", tokens_in=800, tokens_out=200, cost_usd=0.01)
    run.log_screenshot("page.png")          # browser agents
    run.set_output("Lead updated.")
    run.set_cost(0.014)

Every step, tool call, screenshot, cost, and error appears in your AgentQA dashboard seconds later. If your agent raises, the run is recorded as errored with the traceback — and the exception is re-raised untouched.

The SDK never blocks or crashes your agent: steps are sent from a background thread with retries, and network failures degrade to dropped telemetry, not broken automation.

Run a test suite

import agentqa

def my_agent(task_input: str) -> str:
    ...  # your agent, any framework
    return "final answer"

summary = agentqa.run_suite(my_agent, suite_id="...", label="prompt-v4")
print(summary)  # {'suite_run_id': ..., 'cases': 20, 'executed': 20, 'crashed': 0}

Each test case executes locally in your environment — AgentQA never runs your code or holds your credentials. Verdicts and the failure diagnosis appear on the suite run's page as results stream in.

Redact sensitive data

with agentqa.track_run(..., redact=[r"[\w.+-]+@[\w-]+\.[\w.]+"]) as run:
    ...

Redaction runs client-side, before anything leaves your environment. Pass regex patterns or a callable (str) -> str.