Calibrate

Core engine powering Calibrate, a framework for evaluating AI agents which let you move from slow, manual testing to a fast, automated, and repeatable testing process for your entire agent stack:

Text to Text (LLMs): Evaluate the response quality and tool calling of your LLMs for multi-turn conversations and find the find LLM for your agent
Human alignment: Create LLM judges to make your evaluations scalable and reliable with human in the loop.
Speech to Text (STT): Benchmark multiple providers (Google, Sarvam, ElevenLabs and more) on your dataset across 10+ indic languages using metrics optimised for agentic use cases
Text to Speech (TTS): Benchmark generated speech by multiple providers automatically using an Audio LLM Judge across 10+ indic languages
Simulations: Simulate realistic conversations using realistic user personas and scenarios to test failure modes for your agent (including interruptions for voice agents)

Installation

pip install calibrate-agent

Usage

calibrate-agent              # Interactive main menu
calibrate-agent stt          # Benchmark STT providers
calibrate-agent tts          # Benchmark TTS providers
calibrate-agent llm          # Interactive LLM evaluation
calibrate-agent simulations  # Interactive text or voice simulations

CLI Documentation

Contributing

For the web version, see the frontend and backend repositories.

Install development dependencies once (requires uv):

uv sync --extra dev

Running tests

Run the full test suite:

uv run pytest tests/

Pre-commit

Enable the project's git hooks so the pre-commit test runner fires on commits to main:

git config core.hooksPath .githooks

Every contributor needs to run it once.

Citation

If you use Calibrate in your work, please cite it:

@software{calibrate2026,
  author = {Dalmia, Aman and Doshi, Jigar},
  title  = {Calibrate: An open-source framework for evaluating AI agents},
  year   = {2026},
  url    = {https://calibrate.artpark.ai},
  note   = {Source code: \url{https://github.com/ARTPARK-SAHAI-ORG}}
}

License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Metadata

Release files for calibrate-agent 0.0.77

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for calibrate-agent 0.0.77
File	Size	Uploaded
calibrate_agent-0.0.77.tar.gz	18.8 MB	Oct 14, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for calibrate-agent 0.0.77
File	Interpreter	ABI	Platform	Reset
calibrate_agent-0.0.77-py3-none-any.whl 897.1 kB Oct 14, 2026	Python 3	none	any	Details

Total release size: 19.7 MB

Release files / calibrate_agent-0.0.77.tar.gz

Download URL	calibrate_agent-0.0.77.tar.gz
Size	18.8 MB
Tags	Source
SHA-256 checksum How to use checksums	`d8404e52977def3ab3086d104c9583bfb1914d63d30eb6292366c1a4b23586a5`
BLAKE2b-256 checksum How to use checksums	`de22af754127beb28aac793ce5a02766a11c55eef3dce5598deb2661541b1572`
Upload date	Oct 14, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/ARTPARK-SAHAI-ORG/calibrate
Publishing commit: github.com/ARTPARK-SAHAI-ORG/calibrate/tree/b08ee15fe71c5829b138003ffb4f4759335ce644

Workflow

Publishing configuration: github.com/ARTPARK-SAHAI-ORG/calibrate/blob/b08ee15fe71c5829b138003ffb4f4759335ce644/.github/workflows/publish.yml
Publishing logs: github.com/ARTPARK-SAHAI-ORG/calibrate/actions/runs/38612952469/attempts/1

Artifact

Subject: calibrate_agent-0.0.77.tar.gz
SHA-256 checksum: d8404e52977def3ab3086d104c9583bfb1914d63d30eb6292366c1a4b23586a5 Verifying attestations

Release files / calibrate_agent-0.0.77-py3-none-any.whl

Download URL	calibrate_agent-0.0.77-py3-none-any.whl
Size	897.1 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`bb7fc8127e6ace4cc7c4927479744aedeca4e2b9bc436ef60e9036361a7a05bc`
BLAKE2b-256 checksum How to use checksums	`e9e73e2934d09255414bc108b3d1229074b6ba83a129d18dcc7d7f20b68611d9`
Upload date	Oct 14, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/ARTPARK-SAHAI-ORG/calibrate
Publishing commit: github.com/ARTPARK-SAHAI-ORG/calibrate/tree/b08ee15fe71c5829b138003ffb4f4759335ce644

Workflow

Publishing configuration: github.com/ARTPARK-SAHAI-ORG/calibrate/blob/b08ee15fe71c5829b138003ffb4f4759335ce644/.github/workflows/publish.yml
Publishing logs: github.com/ARTPARK-SAHAI-ORG/calibrate/actions/runs/38612952469/attempts/1

Artifact

Subject: calibrate_agent-0.0.77-py3-none-any.whl
SHA-256 checksum: bb7fc8127e6ace4cc7c4927479744aedeca4e2b9bc436ef60e9036361a7a05bc Verifying attestations