Calibrate
Core engine powering Calibrate, a framework for evaluating AI agents which let you move from slow, manual testing to a fast, automated, and repeatable testing process for your entire agent stack:
Text to Text (LLMs): Evaluate the response quality and tool calling of your LLMs for multi-turn conversations and find the find LLM for your agentHuman alignment: Create LLM judges to make your evaluations scalable and reliable with human in the loop.Speech to Text (STT): Benchmark multiple providers (Google, Sarvam, ElevenLabs and more) on your dataset across 10+ indic languages using metrics optimised for agentic use casesText to Speech (TTS): Benchmark generated speech by multiple providers automatically using an Audio LLM Judge across 10+ indic languagesSimulations: Simulate realistic conversations using realistic user personas and scenarios to test failure modes for your agent (including interruptions for voice agents)
Installation
pip install calibrate-agent
Usage
calibrate-agent # Interactive main menu
calibrate-agent stt # Benchmark STT providers
calibrate-agent tts # Benchmark TTS providers
calibrate-agent llm # Interactive LLM evaluation
calibrate-agent simulations # Interactive text or voice simulations
Contributing
For the web version, see the frontend and backend repositories.
Install development dependencies once (requires uv):
uv sync --extra dev
Running tests
Run the full test suite:
uv run pytest tests/
Pre-commit
Enable the project's git hooks so the pre-commit test
runner fires on commits to main:
git config core.hooksPath .githooks
Every contributor needs to run it once.
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file calibrate_agent-0.0.59.tar.gz.
File metadata
- Download URL: calibrate_agent-0.0.59.tar.gz
- Upload date:
- Size: 18.7 MB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
956f57764c41d68383e3750cbc12ad707bfddfc3d9618459cdf62afba980f638
|
|
| MD5 |
ecc24f9be08e0264c55b08973c137cd6
|
|
| BLAKE2b-256 |
1dead26d74c0b9107204e8fe23d8f148e7cbdfdded2bb5c1c619ed25c914e730
|
Provenance
The following attestation bundles were made for calibrate_agent-0.0.59.tar.gz:
Publisher:
publish.yml on ARTPARK-SAHAI-ORG/calibrate
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
calibrate_agent-0.0.59.tar.gz -
Subject digest:
956f57764c41d68383e3750cbc12ad707bfddfc3d9618459cdf62afba980f638 - Sigstore transparency entry: 2136925385
- Sigstore integration time:
-
Permalink:
ARTPARK-SAHAI-ORG/calibrate@a45d84cd56f75e5d41ffdb52ba63054b28ee83d4 -
Branch / Tag:
refs/tags/v0.0.59 - Owner: https://github.com/ARTPARK-SAHAI-ORG
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
publish.yml@a45d84cd56f75e5d41ffdb52ba63054b28ee83d4 -
Trigger Event:
release
-
Statement type:
File details
Details for the file calibrate_agent-0.0.59-py3-none-any.whl.
File metadata
- Download URL: calibrate_agent-0.0.59-py3-none-any.whl
- Upload date:
- Size: 807.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
9ab5dd8c08d2e202e178cb81e0ea7ddfd9745609c517bad78dd3a5d51c1fd3b5
|
|
| MD5 |
f8f558b5dd0ea6ab7e1a3f65f83fd8be
|
|
| BLAKE2b-256 |
79083ae0af1a196348674e95d0680fc03f52086e369499d0c2d93871b5d6d646
|
Provenance
The following attestation bundles were made for calibrate_agent-0.0.59-py3-none-any.whl:
Publisher:
publish.yml on ARTPARK-SAHAI-ORG/calibrate
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
calibrate_agent-0.0.59-py3-none-any.whl -
Subject digest:
9ab5dd8c08d2e202e178cb81e0ea7ddfd9745609c517bad78dd3a5d51c1fd3b5 - Sigstore transparency entry: 2136925407
- Sigstore integration time:
-
Permalink:
ARTPARK-SAHAI-ORG/calibrate@a45d84cd56f75e5d41ffdb52ba63054b28ee83d4 -
Branch / Tag:
refs/tags/v0.0.59 - Owner: https://github.com/ARTPARK-SAHAI-ORG
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
publish.yml@a45d84cd56f75e5d41ffdb52ba63054b28ee83d4 -
Trigger Event:
release
-
Statement type:
