Skip to main content

ToolRate Python SDK

Python client for the ToolRate API — the reliability oracle for AI agents.

Installation

Recommended (modern & fastest):

# Install uv (one-time)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Add ToolRate to your project
uv add toolrate

Alternative (without uv):

python3 -m venv .venv
source .venv/bin/activate
pip install toolrate

Note: If you see a PEP 668 "externally-managed-environment" error with plain pip, that is because of Homebrew Python. Use one of the methods above instead.

Using poetry, pdm, or rye? poetry add toolrate (or pdm add, rye add) works too.

Quick start — one line of code

The guard function wraps any tool call with automatic reliability checking:

from toolrate import ToolRate, guard

client = ToolRate(api_key="nf_live_...")

# Wrap any tool call — assesses before, reports after, automatically
result = guard(client, "https://api.openai.com/v1/chat/completions",
               lambda: openai.chat.completions.create(model="gpt-4", messages=[...]))

That's it. ToolRate will:

  1. Check the tool's reliability score before calling
  2. Execute the tool call
  3. Report success/failure back (building the data moat)
  4. Classify errors automatically

Auto-fallback

When a tool fails, automatically try alternatives:

result = guard(
    client,
    "https://api.openai.com/v1/chat/completions",
    lambda: openai.chat.completions.create(model="gpt-4", messages=msgs),
    fallbacks=[
        ("https://api.anthropic.com/v1/messages",
         lambda: anthropic.messages.create(model="claude-sonnet-4-20250514", messages=msgs)),
        ("https://api.groq.com/openai/v1/chat/completions",
         lambda: groq.chat.completions.create(model="llama-3.3-70b", messages=msgs)),
    ],
    min_score=50,  # Skip tools scoring below 50
)

Decorator

from toolrate import ToolRate, toolrate_guard

client = ToolRate(api_key="nf_live_...")

@toolrate_guard(client, "https://api.stripe.com/v1/charges")
def charge_customer(amount, currency):
    return stripe.Charge.create(amount=amount, currency=currency)

# Every call is now automatically assessed + reported
charge_customer(1000, "usd")

Journey tracking

Report fallback patterns to power hidden gem discovery:

# First attempt fails
client.report("https://api.sendgrid.com/v3/mail/send",
    success=False, error_category="rate_limit",
    session_id="session-123", attempt_number=1)

# Fallback succeeds
client.report("https://api.resend.com/emails",
    success=True, latency_ms=180,
    session_id="session-123", attempt_number=2,
    previous_tool="https://api.sendgrid.com/v3/mail/send")

Discovery

Find hidden gems and fallback chains based on real agent behavior:

# Tools that shine as fallbacks
gems = client.discover_hidden_gems(category="email")

# What to try when SendGrid fails
chain = client.discover_fallback_chain("https://api.sendgrid.com/v3/mail/send")

Direct API usage

from toolrate import ToolRate

client = ToolRate(api_key="nf_live_...")

# Assess
result = client.assess("https://api.openai.com/v1/chat/completions",
                        context="customer support chatbot")
print(result["reliability_score"])      # 89.0
print(result["predicted_failure_risk"]) # "low"
print(result["common_pitfalls"])        # ["timeout (8% of failures)"]
print(result["top_alternatives"])       # [{"tool": "...", "score": 90}]

# Report
client.report("https://api.openai.com/v1/chat/completions",
              success=True, latency_ms=2500)

client.close()

Async support

AsyncToolRate has the same interface — all methods are async.

from toolrate import AsyncToolRate

async with AsyncToolRate(api_key="nf_live_...") as client:
    result = await client.assess("https://api.openai.com/v1/chat/completions")

Metadata

Release files for toolrate 0.6.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for toolrate 0.6.1
File Size Uploaded Provenance Actions
toolrate-0.6.1.tar.gz 16.6 kB Without provenance Details for toolrate-0.6.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for toolrate 0.6.1
File Interpreter ABI Platform Actions
toolrate-0.6.1-py3-none-any.whl Python 3 none any Details for toolrate-0.6.1-py3-none-any.whl

Total release size: 29.6 kB

Release files / toolrate-0.6.1.tar.gz

Download URL toolrate-0.6.1.tar.gz
Size 16.6 kB
Tags Source
SHA-256 checksum
How to use checksums
093eeafccb6ae4e0121cfca9f2e20025c6df6184e2d02795fb9dae94c9409e5b
BLAKE2b-256 checksum
How to use checksums
4328ab65176b3f57002c5a24ea6529feb64f297c5311224a751c193130918223
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.14.3

Release files / toolrate-0.6.1-py3-none-any.whl

Download URL toolrate-0.6.1-py3-none-any.whl
Size 13.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
c52367d5672cf2bd162862e5e539facdfa08555b83378158fd9017e7c7c463ea
BLAKE2b-256 checksum
How to use checksums
16571f2e5eef31b87ffd90e32c663f334704c1b8b96825fc22efe43550ca1d24
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.14.3

Security for toolrate 0.6.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page