Calibrate

Core engine powering Calibrate, a framework for evaluating AI agents which let you move from slow, manual testing to a fast, automated, and repeatable testing process for your entire agent stack:

• Text to Text (LLMs): Evaluate the response quality and tool calling of your LLMs for multi-turn conversations and find the find LLM for your agent
• Human alignment: Create LLM judges to make your evaluations scalable and reliable with human in the loop.
• Speech to Text (STT): Benchmark multiple providers (Google, Sarvam, ElevenLabs and more) on your dataset across 10+ indic languages using metrics optimised for agentic use cases
• Text to Speech (TTS): Benchmark generated speech by multiple providers automatically using an Audio LLM Judge across 10+ indic languages
• Simulations: Simulate realistic conversations using realistic user personas and scenarios to test failure modes for your agent (including interruptions for voice agents)

Installation

pip install calibrate-agent

Usage

calibrate-agent              # Interactive main menu
calibrate-agent stt          # Benchmark STT providers
calibrate-agent tts          # Benchmark TTS providers
calibrate-agent llm          # Interactive LLM evaluation
calibrate-agent simulations  # Interactive text or voice simulations

• CLI Documentation

Contributing

For the web version, see the frontend and backend repositories.

Install development dependencies once (requires uv):

uv sync --extra dev

Running tests

Run the full test suite:

uv run pytest tests/

Pre-commit

Enable the project's git hooks so the pre-commit test runner fires on commits to main:

git config core.hooksPath .githooks

Every contributor needs to run it once.

Citation

If you use Calibrate in your work, please cite it:

@software{calibrate2026,
  author = {Dalmia, Aman and Doshi, Jigar},
  title  = {Calibrate: An open-source framework for evaluating AI agents},
  year   = {2026},
  url    = {https://calibrate.artpark.ai},
  note   = {Source code: \url{https://github.com/ARTPARK-SAHAI-ORG}}
}

License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.