Calibrate

Core engine powering Calibrate, a framework for evaluating AI agents which let you move from slow, manual testing to a fast, automated, and repeatable testing process for your entire agent stack:

Text to Text (LLMs): Evaluate the response quality and tool calling of your LLMs for multi-turn conversations and find the find LLM for your agent
Human alignment: Create LLM judges to make your evaluations scalable and reliable with human in the loop.
Speech to Text (STT): Benchmark multiple providers (Google, Sarvam, ElevenLabs and more) on your dataset across 10+ indic languages using metrics optimised for agentic use cases
Text to Speech (TTS): Benchmark generated speech by multiple providers automatically using an Audio LLM Judge across 10+ indic languages
Simulations: Simulate realistic conversations using realistic user personas and scenarios to test failure modes for your agent (including interruptions for voice agents)

Installation

pip install calibrate-agent

Usage

calibrate-agent              # Interactive main menu
calibrate-agent stt          # Benchmark STT providers
calibrate-agent tts          # Benchmark TTS providers
calibrate-agent llm          # Interactive LLM evaluation
calibrate-agent simulations  # Interactive text or voice simulations

CLI Documentation

Contributing

For the web version, see the frontend and backend repositories.

Install development dependencies once (requires uv):

uv sync --extra dev

Running tests

Run the full test suite:

uv run pytest tests/

Pre-commit

Enable the project's git hooks so the pre-commit test runner fires on commits to main:

git config core.hooksPath .githooks

Every contributor needs to run it once.

Citation

If you use Calibrate in your work, please cite it:

@software{calibrate2026,
  author = {Dalmia, Aman and Doshi, Jigar},
  title  = {Calibrate: An open-source framework for evaluating AI agents},
  year   = {2026},
  url    = {https://calibrate.artpark.ai},
  note   = {Source code: \url{https://github.com/ARTPARK-SAHAI-ORG}}
}

License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Metadata

Release files for calibrate-agent 0.0.76

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for calibrate-agent 0.0.76
File	Size	Uploaded
calibrate_agent-0.0.76.tar.gz	18.8 MB	Sep 17, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for calibrate-agent 0.0.76
File	Interpreter	ABI	Platform	Reset
calibrate_agent-0.0.76-py3-none-any.whl 896.7 kB Sep 17, 2026	Python 3	none	any	Details

Total release size: 19.7 MB

Release files / calibrate_agent-0.0.76.tar.gz

Download URL	calibrate_agent-0.0.76.tar.gz
Size	18.8 MB
Tags	Source
SHA-256 checksum How to use checksums	`75067e1958e5bdd64c92652bb7467f02c0774a32c527fd081f4bcac0b3f261a7`
BLAKE2b-256 checksum How to use checksums	`58efccedbcbbdef7794567a7abc74e3650a4a82b110ab78e6c4960cb307f9fbe`
Upload date	Sep 17, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/ARTPARK-SAHAI-ORG/calibrate
Publishing commit: github.com/ARTPARK-SAHAI-ORG/calibrate/tree/c628aec251bcb555f3f114f55187879d6dd14cb1

Workflow

Publishing configuration: github.com/ARTPARK-SAHAI-ORG/calibrate/blob/c628aec251bcb555f3f114f55187879d6dd14cb1/.github/workflows/publish.yml
Publishing logs: github.com/ARTPARK-SAHAI-ORG/calibrate/actions/runs/35239930638/attempts/1

Artifact

Subject: calibrate_agent-0.0.76.tar.gz
SHA-256 checksum: 75067e1958e5bdd64c92652bb7467f02c0774a32c527fd081f4bcac0b3f261a7 Verifying attestations

Release files / calibrate_agent-0.0.76-py3-none-any.whl

Download URL	calibrate_agent-0.0.76-py3-none-any.whl
Size	896.7 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`eebb7e98a95f21218ae5a589b16ba06d25326272b6d4bda73c62abec249fdf9c`
BLAKE2b-256 checksum How to use checksums	`254d167d2d0c8c53ea4867bac9cc695f807c9fd1267fad74f2bdee695f1adb81`
Upload date	Sep 17, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/ARTPARK-SAHAI-ORG/calibrate
Publishing commit: github.com/ARTPARK-SAHAI-ORG/calibrate/tree/c628aec251bcb555f3f114f55187879d6dd14cb1

Workflow

Publishing configuration: github.com/ARTPARK-SAHAI-ORG/calibrate/blob/c628aec251bcb555f3f114f55187879d6dd14cb1/.github/workflows/publish.yml
Publishing logs: github.com/ARTPARK-SAHAI-ORG/calibrate/actions/runs/35239930638/attempts/1

Artifact

Subject: calibrate_agent-0.0.76-py3-none-any.whl
SHA-256 checksum: eebb7e98a95f21218ae5a589b16ba06d25326272b6d4bda73c62abec249fdf9c Verifying attestations