toolgauntlet 0.1.0

Deterministic reliability testing for tool-using AI agents

ToolGauntlet

ToolGauntlet by HiLo Labs is a Python package and CLI for deterministic reliability testing of tool-using AI agents.

Install

pip install toolgauntlet
toolgauntlet --version

The distribution, primary CLI, and public Python import are all toolgauntlet. The prerelease agent-chaos commands and agent_chaos import remain available as compatibility aliases.

Quick start (CLI)

Generate a complete local baseline with no API key:

toolgauntlet quickstart --out-dir toolgauntlet-quickstart

This writes JSON, Markdown, and HTML baseline reports plus a copyable next-step regression command. It uses the built-in deterministic adapter with safe logging enabled and refuses to overwrite its files unless --force is supplied.

Build a custom workflow manually:

toolgauntlet list-suites
toolgauntlet init-suite --id my_suite_v1 --out-dir suites
toolgauntlet run --suite ecommerce_refunds_v1 --runs 10 --out report.json
toolgauntlet report --in report.json --format md --out report.md
toolgauntlet report --in report.json --format html --out report.html

Use a saved baseline as a release gate:

toolgauntlet run --suite ecommerce_refunds_v1 --runs 25 --out baseline.json
toolgauntlet run \
  --suite ecommerce_refunds_v1 \
  --runs 25 \
  --baseline baseline.json \
  --fail-on-regression \
  --regression-tolerance 1.0 \
  --out candidate.json

See the copyable CI workflow in docs/ci-gate-example.md and a sample report in docs/sample-reports/ecommerce_refunds_v1.md.

Before release, run the same local gate used for MVP validation:

scripts/release_gate.sh

Verify the configured external launch stage (live page now; registries as the stage advances):

python scripts/check_launch_readiness.py

Signed pack workflow:

export TOOLGAUNTLET_PACK_KEY=your-shared-signing-key
toolgauntlet sign-pack --suite suites/my_suite_v1
toolgauntlet verify-pack --suite suites/my_suite_v1
toolgauntlet run --suite suites/my_suite_v1 --verify-pack --out report.json

OpenTelemetry traces (optional):

pip install "toolgauntlet[otel]"
# Requires an OTLP/HTTP collector reachable at the default endpoint,
# or pass --otel-endpoint with your collector URL.
toolgauntlet run --suite ecommerce_refunds_v1 --runs 5 --otel-enabled --otel-metrics-enabled

Quick start (Python)

from toolgauntlet import SuiteConfig, run_suite
from toolgauntlet.adapters import deterministic_demo_agent

report = run_suite(
    agent=deterministic_demo_agent,
    suite_path="ecommerce_refunds_v1",
    config=SuiteConfig(runs=10, concurrency=2),
)

report.save_json("report.json")
report.save_markdown("report.md")

OpenAI-compatible adapter

from toolgauntlet import SuiteConfig, run_suite
from toolgauntlet.adapters import OpenAICompatibleAdapterConfig, make_openai_tool_loop_adapter

agent = make_openai_tool_loop_adapter(
    OpenAICompatibleAdapterConfig(
        model="gpt-4o-mini",
        base_url="https://api.openai.com",
    )
)

report = run_suite(
    agent=agent,
    suite_path="ecommerce_refunds_v1",
    config=SuiteConfig(runs=5, concurrency=1),
)

Additional adapter helpers

• make_async_adapter: use async agent callables with the synchronous harness interface.
• make_langchain_like_adapter: adapt invoke/ainvoke-style runnables.

See examples in examples/adapters/.

Budget-as-Code Proxy

Start and validate:

toolgauntlet-proxy validate-policy --policy examples/budget_proxy/policy.example.yaml
toolgauntlet-proxy serve \
  --host 127.0.0.1 \
  --port 8080 \
  --policy examples/budget_proxy/policy.example.yaml \
  --ledger logs/usage-ledger.sqlite

One-command local stack (proxy + redis + optional otel/prometheus):

docker compose --env-file .env.proxy -f docker-compose.proxy.yml up --build

See full guide: docs/compose-deployment.md

Build and preview site pages locally:

pip install -e ".[dev,site]"
toolgauntlet build-site --pages-root site/pages/toolgauntlet --out-dir site/dist/toolgauntlet
python -m http.server 8787 --directory site/dist

Key endpoints:

• GET /healthz
• GET /metrics
• GET /v1/models
• POST /v1/chat/completions
• POST /v1/responses
• POST /admin/reload-policy

Common environment variables:

• OPENAI_API_KEY or BUDGET_UPSTREAM_API_KEY
• BUDGET_PROXY_API_KEY
• BUDGET_POLICY_PATH
• BUDGET_AUDIT_LOG_PATH
• BUDGET_LEDGER_PATH
• BUDGET_PROJECT_HEADER
• BUDGET_PROJECT_API_KEY_HEADER
• BUDGET_METRIC_NAMESPACE
• BUDGET_RATE_LIMIT_BACKEND (memory or redis)
• BUDGET_REDIS_URL (required when backend is redis)
• BUDGET_REDIS_KEY_PREFIX
• BUDGET_OTEL_ENABLED
• BUDGET_OTEL_METRICS_ENABLED
• BUDGET_OTEL_SERVICE_NAME
• BUDGET_OTEL_ENDPOINT

To enable distributed rate limiting across proxy instances, install Redis support:

pip install "toolgauntlet[redis]"

Included suites

• ecommerce_refunds_v1
• support_triage_v1
• travel_booking_v1

Documentation

• docs index: docs/README.md
• API reference: docs/api-reference.md
• proxy operations: docs/proxy-operations.md
• compose deployment: docs/compose-deployment.md
• site preview: docs/site-preview.md
• CI gate example: docs/ci-gate-example.md
• commercial pack readiness: docs/commercial-pack-readiness.md
• customer pack handoff: docs/customer-pack-handoff.md
• launch operations checklist: docs/launch-operations-checklist.md
• AI venture operating plan: docs/venture-operating-plan.md
• AI venture operator: docs/venture-operator.md
• release checklist: docs/release-checklist.md
• versioning policy: docs/versioning-policy.md
• entitlement model: docs/entitlement-model.md
• support SLA: docs/support-sla.md

Marketing and launch materials

• materials index: marketing/README.md
• website copy: site/pages/toolgauntlet/
• pilot sales path: site/pages/toolgauntlet/pilot.md and docs/pilot-intake.md

Backlog and roadmap

• product backlog: TODO.md

Release workflows

GitHub Actions workflows included:

• .github/workflows/ci.yml
• .github/workflows/launch-readiness.yml
• .github/workflows/venture-ops.yml
• .github/workflows/lead-intake.yml
• .github/workflows/release.yml

Publishing uses trusted publishing via pypa/gh-action-pypi-publish.