calibrate-agent 0.0.49

An open-source evaluation framework for voice agents

Calibrate

CLI for Calibrate: an AI evaluation platform for non-profits

With Calibrate, you can move from slow, manual testing to a fast, automated, and repeatable testing process for your entire agent stack:

• Text to Text (LLMs): Evaluate the response quality and tool calling of your LLMs for multi-turn conversations and find the find LLM for your agent
• Human alignment: Create LLM judges to make your evaluations scalable and reliable with human in the loop.
• Speech to Text (STT): Benchmark multiple providers (Google, Sarvam, ElevenLabs and more) on your dataset across 10+ indic languages using metrics optimised for agentic use cases
• Text to Speech (TTS): Benchmark generated speech by multiple providers automatically using an Audio LLM Judge across 10+ indic languages
• Simulations: Simulate realistic conversations using realistic user personas and scenarios to test failure modes for your agent (including interruptions for voice agents)

Calibrate is built on top of pipecat, a framework for building agents.

Installation

pip install calibrate-agent

Usage

calibrate              # Interactive main menu
calibrate stt          # Benchmark STT providers
calibrate tts          # Benchmark TTS providers
calibrate llm          # Interactive LLM evaluation
calibrate simulations  # Interactive text or voice simulations

• CLI Documentation

Contributing

Install development dependencies once (requires uv):

uv sync --extra dev

Running tests

Run the full test suite:

uv run pytest tests/

Pre-commit

Enable the project's git hooks so the pre-commit test runner fires on commits to main:

git config core.hooksPath .githooks

Every contributor needs to run it once.

License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.