ToolRate Python SDK
Python client for the ToolRate API — the reliability oracle for AI agents.
Installation
Recommended (modern & fastest):
# Install uv (one-time)
curl -LsSf https://astral.sh/uv/install.sh | sh
# Add ToolRate to your project
uv add toolrate
Alternative (without uv):
python3 -m venv .venv
source .venv/bin/activate
pip install toolrate
Note: If you see a
PEP 668"externally-managed-environment" error with plainpip, that is because of Homebrew Python. Use one of the methods above instead.
Using poetry, pdm, or rye? poetry add toolrate (or pdm add, rye add) works too.
Quick start — one line of code
The guard function wraps any tool call with automatic reliability checking:
from toolrate import ToolRate, guard
client = ToolRate(api_key="nf_live_...")
# Wrap any tool call — assesses before, reports after, automatically
result = guard(client, "https://api.openai.com/v1/chat/completions",
lambda: openai.chat.completions.create(model="gpt-4", messages=[...]))
That's it. ToolRate will:
- Check the tool's reliability score before calling
- Execute the tool call
- Report success/failure back (building the data moat)
- Classify errors automatically
Auto-fallback
When a tool fails, automatically try alternatives:
result = guard(
client,
"https://api.openai.com/v1/chat/completions",
lambda: openai.chat.completions.create(model="gpt-4", messages=msgs),
fallbacks=[
("https://api.anthropic.com/v1/messages",
lambda: anthropic.messages.create(model="claude-sonnet-4-20250514", messages=msgs)),
("https://api.groq.com/openai/v1/chat/completions",
lambda: groq.chat.completions.create(model="llama-3.3-70b", messages=msgs)),
],
min_score=50, # Skip tools scoring below 50
)
Decorator
from toolrate import ToolRate, toolrate_guard
client = ToolRate(api_key="nf_live_...")
@toolrate_guard(client, "https://api.stripe.com/v1/charges")
def charge_customer(amount, currency):
return stripe.Charge.create(amount=amount, currency=currency)
# Every call is now automatically assessed + reported
charge_customer(1000, "usd")
Journey tracking
Report fallback patterns to power hidden gem discovery:
# First attempt fails
client.report("https://api.sendgrid.com/v3/mail/send",
success=False, error_category="rate_limit",
session_id="session-123", attempt_number=1)
# Fallback succeeds
client.report("https://api.resend.com/emails",
success=True, latency_ms=180,
session_id="session-123", attempt_number=2,
previous_tool="https://api.sendgrid.com/v3/mail/send")
Discovery
Find hidden gems and fallback chains based on real agent behavior:
# Tools that shine as fallbacks
gems = client.discover_hidden_gems(category="email")
# What to try when SendGrid fails
chain = client.discover_fallback_chain("https://api.sendgrid.com/v3/mail/send")
Direct API usage
from toolrate import ToolRate
client = ToolRate(api_key="nf_live_...")
# Assess
result = client.assess("https://api.openai.com/v1/chat/completions",
context="customer support chatbot")
print(result["reliability_score"]) # 89.0
print(result["predicted_failure_risk"]) # "low"
print(result["common_pitfalls"]) # ["timeout (8% of failures)"]
print(result["top_alternatives"]) # [{"tool": "...", "score": 90}]
# Report
client.report("https://api.openai.com/v1/chat/completions",
success=True, latency_ms=2500)
client.close()
Async support
AsyncToolRate has the same interface — all methods are async.
from toolrate import AsyncToolRate
async with AsyncToolRate(api_key="nf_live_...") as client:
result = await client.assess("https://api.openai.com/v1/chat/completions")
Metadata
Release files for toolrate 0.6.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| toolrate-0.6.1.tar.gz | 16.6 kB | Without provenance | Details for toolrate-0.6.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| toolrate-0.6.1-py3-none-any.whl | Python 3 | none | any | Details for toolrate-0.6.1-py3-none-any.whl |
Total release size: 29.6 kB
Release files / toolrate-0.6.1.tar.gz
| Download URL | toolrate-0.6.1.tar.gz |
|---|---|
| Size | 16.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
093eeafccb6ae4e0121cfca9f2e20025c6df6184e2d02795fb9dae94c9409e5b
|
|
BLAKE2b-256 checksum How to use checksums |
4328ab65176b3f57002c5a24ea6529feb64f297c5311224a751c193130918223
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.14.3
|
Release files / toolrate-0.6.1-py3-none-any.whl
| Download URL | toolrate-0.6.1-py3-none-any.whl |
|---|---|
| Size | 13.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
c52367d5672cf2bd162862e5e539facdfa08555b83378158fd9017e7c7c463ea
|
|
BLAKE2b-256 checksum How to use checksums |
16571f2e5eef31b87ffd90e32c663f334704c1b8b96825fc22efe43550ca1d24
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.14.3
|
Security for toolrate 0.6.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.