Calibrate Python Library
The Calibrate Python SDK provides convenient, typed access to the Calibrate
agent evaluation platform API. Authenticate with an API key (sk_…) scoped
to your organization to run agent tests and poll their results.
Table of Contents
- Documentation
- Installation
- Pre Requisites
- Reference
- Quick Start
- Usage
- Environments
- Async Client
- Exception Handling
- Advanced
Documentation
API reference documentation is available here.
Installation
pip install calibrate-sdk
Pre-requisites
You need to have a Calibrate API key. You can get the API key from the UI.
Reference
A full reference for this library is available here.
Quick Start
from calibrate import Calibrate
client = Calibrate(api_key="sk_...")
# List the agents in your organization
agents = client.agents.list()
# Kick off a test run for an agent and poll for the result
run = client.agent_tests.run(agent_uuid="<agent-uuid>")
result = client.agent_tests.get_run(task_id=run.task_id)
Usage
Instantiate and use the client with the following:
from calibrate import Calibrate
client = Calibrate(
api_key="<value>",
)
client.agents.resolve(
names=[
"my-agent",
"support-bot"
],
)
Environments
This SDK allows you to configure different environments for API requests.
from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment
client = Calibrate(
environment=CalibrateEnvironment.DEFAULT,
)
Async Client
The SDK also exports an async client so that you can make non-blocking calls to our API. Note that if you are constructing an Async httpx client class to pass into this client, use httpx.AsyncClient() instead of httpx.Client() (e.g. for the httpx_client parameter of this client).
import asyncio
from calibrate import AsyncCalibrate
client = AsyncCalibrate(
api_key="<value>",
)
async def main() -> None:
await client.agents.resolve(
names=[
"my-agent",
"support-bot"
],
)
asyncio.run(main())
Exception Handling
When the API returns a non-success status code (4xx or 5xx response), a subclass of the following error will be thrown.
from calibrate.core.api_error import ApiError
try:
client.agents.resolve(...)
except ApiError as e:
print(e.status_code)
print(e.body)
Advanced
Access Raw Response Data
The SDK provides access to raw response data, including headers, through the .with_raw_response property.
The .with_raw_response property returns a "raw" client that can be used to access the .headers and .data attributes.
from calibrate import Calibrate
client = Calibrate(...)
response = client.agents.with_raw_response.resolve(...)
print(response.headers) # access the response headers
print(response.status_code) # access the response status code
print(response.data) # access the underlying object
Retries
The SDK is instrumented with automatic retries with exponential backoff. A request will be retried as long as the request is deemed retryable and the number of retry attempts has not grown larger than the configured retry limit (default: 2).
Which status codes are retried depends on the retryStatusCodes generator configuration:
legacy (current default): retries on
recommended: retries on
- 408 (Timeout)
- 409 (Conflict)
- 429 (Too Many Requests)
- 502 (Bad Gateway)
- 503 (Service Unavailable)
- 504 (Gateway Timeout)
Use the max_retries request option to configure this behavior.
client.agents.resolve(..., request_options={
"max_retries": 1
})
Timeouts
The SDK defaults to a 60 second timeout. You can configure this with a timeout option at the client or request level.
from calibrate import Calibrate
client = Calibrate(..., timeout=20.0)
# Override timeout for a specific method
client.agents.resolve(..., request_options={
"timeout_in_seconds": 1
})
Custom Client
You can override the httpx client to customize it for your use-case. Some common use-cases include support for proxies
and transports.
import httpx
from calibrate import Calibrate
client = Calibrate(
...,
httpx_client=httpx.Client(
proxy="http://my.test.proxy.example.com",
transport=httpx.HTTPTransport(local_address="0.0.0.0"),
),
)
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file calibrate_sdk-0.0.13.tar.gz.
File metadata
- Download URL: calibrate_sdk-0.0.13.tar.gz
- Upload date:
- Size: 47.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
poetry/1.5.1 CPython/3.10.20 Linux/6.17.0-1018-azure
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
21626894d27ec2c39a516286154be042898b7737403092b353d457e59392e1fd
|
|
| MD5 |
5d5181ec5fb99446afcb8262f68c1021
|
|
| BLAKE2b-256 |
4f38d52494256c5bea4429994260cfa1af9879fb6067bc3a43382e9bcacf670e
|
File details
Details for the file calibrate_sdk-0.0.13-py3-none-any.whl.
File metadata
- Download URL: calibrate_sdk-0.0.13-py3-none-any.whl
- Upload date:
- Size: 78.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
poetry/1.5.1 CPython/3.10.20 Linux/6.17.0-1018-azure
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
fc13a77d1cc2dc3f8cf4edc0724e4fd89c2ced74df92c820449fb15bfcda82c7
|
|
| MD5 |
9b74d0071345a95dbda001937e81ad00
|
|
| BLAKE2b-256 |
681292ccd907ae2bc4a3dc4a09b6adafd9445e97dbbe49a9c6a54e10d688775d
|
