benchpress-agent

The reliability layer for AI agents with write access. Benchpress is a task-agnostic control loop you wrap around any tool layer. It reads the workspace's rules before deciding what "done" means. It picks the right record among look-alikes and locks the rest in a deny-list enforced in code. It plans only the writes the definition of done needs, gates every one, and reads every write back. Status comes from provider state, never from an HTTP 200. Every run ends with an auditable receipt.

pip install benchpress-agent        # or: uv add benchpress-agent

Python 3.12+. Runtime dependencies: httpx, pydantic.

Quickstart

import asyncio
import benchpress

async def execute_tool(tool_name: str, tool_input: dict) -> dict:
    # tool_name == "provider_api"
    # tool_input == {"provider": "stripe", "method": "POST", "path": "/v1/customers/cus_123",
    #                "query": {...}, "body": {...}}
    # Call your real API (or a sandbox/twin) and return:
    return {"ok": True, "status_code": 200, "body": {...}}

agent = benchpress.wrap("deepseek-v4-pro", execute_tool, providers=["hubspot", "stripe", "slack", "gmail"])

result = asyncio.run(agent.run(
    "Acme asked for renewal notices to go to ap@acme.example. Update billing and the CRM, "
    "and keep the account owner in the loop. Do not send external mail.",
    trace_dir="runs/acme",            # writes receipt.json
))
print(result.status)                  # "completed" | "partial" | "escalated", from read-back evidence only
print(result.context.refusals)        # every write the gate refused, with the rule that refused it

• model is a model id (DEEPSEEK_API_KEY / BENCHPRESS_API_KEY + BENCHPRESS_API_BASE for any OpenAI-compatible endpoint, ANTHROPIC_API_KEY for claude-*), a benchpress.ModelConfig, or None to read BENCHPRESS_MODEL.
• executor is an async execute_tool(tool_name, tool_input) function or any object exposing one.
• Built-in playbooks cover Slack, Gmail, HubSpot and Stripe; pass playbooks= for your own systems.
• transport= swaps the model wire protocol (bring your own client, or a scripted one in tests).
• agent.run_sync(...) for synchronous callers.

A real-app gateway for Slack, Gmail, HubSpot and Stripe ships in benchpress.realapp (gateway_from_env(providers)), and the CLI runs a request end to end:

benchpress run --providers hubspot,stripe --prompt "..." --trace-dir runs/demo
benchpress receipt runs/demo --html

The loop

P0 orient → P1 policy sweep → P2 resolve (lock look-alikes) → P3 definition of done → P4 plan → P5 execute through the gate + read-back → P6 verify (+ one repair round) → P7 deliver

Evidence

Benchpress was built for the Multi-App AI Agent Hackathon (2026-09-13) and is measured against ArgaBench's hardest published scenario, graded by ArgaBench's verifier on the published seed rebuilt locally and on real Slack, Gmail, HubSpot and Stripe (test mode). It makes no claim about the official leaderboard. Methods, results, failures and cost are in the repository: README · reports · disclosure.

Status

Alpha (0.x). The public API is benchpress.wrap, Benchpress.run, run_trial, TrialResult, ModelConfig, Ablations, Gate. Expect additions (MCP guard, policy packs, Rehearse) in minor releases; see the roadmap.

Apache-2.0 · Built by Raj Karia