calibrate-agent 0.0.3

An open-source evaluation framework for voice agents

Calibrate

An open-source evaluation framework for voice agents.

With Calibrate, you can move from slow, manual testing to a fast, automated, and repeatable testing process for your entire voice agent stack:

• Speech to Text (STT): Benchmark multiple providers (Google, Sarvam, ElevenLabs and more) on your dataset across 10+ indic languages using metrics optimised for agentic use cases
• Text to Speech (TTS): Benchmark generated speech by multiple providers automatically using an Audio LLM Judge across 10+ indic languages
• Text to Text (LLMs): Evaluate the response quality and tool calling of your LLMs for multi-turn conversations and find the find LLM for your agent
• Simulations: Simulate realistic conversations using realistic user personas and scenarios to test failure modes for your agent including interruptions

Calibrate is built on top of pipecat, a framework for building voice agents.

Installation

pip install calibrate-agent

Usage

calibrate              # Interactive main menu
calibrate stt          # Benchmark STT providers
calibrate tts          # Benchmark TTS providers
calibrate llm          # Interactive LLM evaluation
calibrate simulations  # Interactive text or voice simulations

• CLI Documentation

License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.