Calibrate

WhatsApp codecov CC BY-SA 4.0

Core engine powering Calibrate, a framework for evaluating AI agents which let you move from slow, manual testing to a fast, automated, and repeatable testing process for your entire agent stack:

  • Text to Text (LLMs): Evaluate the response quality and tool calling of your LLMs for multi-turn conversations and find the find LLM for your agent
  • Human alignment: Create LLM judges to make your evaluations scalable and reliable with human in the loop.
  • Speech to Text (STT): Benchmark multiple providers (Google, Sarvam, ElevenLabs and more) on your dataset across 10+ indic languages using metrics optimised for agentic use cases
  • Text to Speech (TTS): Benchmark generated speech by multiple providers automatically using an Audio LLM Judge across 10+ indic languages
  • Simulations: Simulate realistic conversations using realistic user personas and scenarios to test failure modes for your agent (including interruptions for voice agents)

Installation

pip install calibrate-agent

Usage

calibrate-agent              # Interactive main menu
calibrate-agent stt          # Benchmark STT providers
calibrate-agent tts          # Benchmark TTS providers
calibrate-agent llm          # Interactive LLM evaluation
calibrate-agent simulations  # Interactive text or voice simulations

Contributing

For the web version, see the frontend and backend repositories.

Install development dependencies once (requires uv):

uv sync --extra dev

Running tests

Run the full test suite:

uv run pytest tests/

Pre-commit

Enable the project's git hooks so the pre-commit test runner fires on commits to main:

git config core.hooksPath .githooks

Every contributor needs to run it once.

License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

CC BY-SA 4.0

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.0.75)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for aman-artpark from gravatar.com aman-artpark

Credits

Author: Aman Dalmia

License expression

CC-BY-SA-4.0
View SPDX License List

Requires

Python >=3.11

Provides Extra

dev

Tags

voice-agents evaluation benchmarking stt tts llm simulation pipecat

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

calibrate_agent-0.0.74.tar.gz (18.8 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

calibrate_agent-0.0.74-py3-none-any.whl (895.3 kB view details)

Uploaded Python 3

File details

Details for the file calibrate_agent-0.0.74.tar.gz.

File metadata

  • Download URL: calibrate_agent-0.0.74.tar.gz
  • Upload date:
  • Size: 18.8 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for calibrate_agent-0.0.74.tar.gz
Algorithm Hash digest
SHA256 281f8ab48a5f8721d997e9f0102b292d9c5c1a9cc191b32c780ee7ab4f0c1362
MD5 ce57056e06389c06e54a688b3ff7e9b6
BLAKE2b-256 8436c9eb09a94570b7cff9f39ddb883f75239bfc168016b44faeddef142b2be6

See more details on using hashes here.

Provenance

The following attestation bundles were made for calibrate_agent-0.0.74.tar.gz:

Publisher: publish.yml on ARTPARK-SAHAI-ORG/calibrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file calibrate_agent-0.0.74-py3-none-any.whl.

File metadata

File hashes

Hashes for calibrate_agent-0.0.74-py3-none-any.whl
Algorithm Hash digest
SHA256 87c1af58cafb62de72f2d3f6eaec51f1c5648aca0d917670e08a75f954d64a65
MD5 dc6114b3a9065007f642ce479d1cbfa2
BLAKE2b-256 b4e2f69d3857f19aa055cd817cfb7ecb973f7b60eb81ca81f78c0cb29b013f02

See more details on using hashes here.

Provenance

The following attestation bundles were made for calibrate_agent-0.0.74-py3-none-any.whl:

Publisher: publish.yml on ARTPARK-SAHAI-ORG/calibrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.0.75

2 files

This release

0.0.74 This release

2 files

0.0.73

2 files

0.0.72

2 files

0.0.71

2 files

0.0.70

2 files

0.0.69

2 files

0.0.68

2 files

0.0.67

2 files

0.0.66

2 files

0.0.65

2 files

0.0.64

2 files

0.0.63

2 files

0.0.62

2 files

0.0.61

2 files

0.0.60

2 files

0.0.59

2 files

0.0.58

2 files

0.0.57

2 files

0.0.56

2 files

0.0.55

2 files

0.0.54

2 files

0.0.53

2 files

0.0.52

2 files

0.0.51

2 files

0.0.50

2 files

0.0.49

2 files

0.0.48

2 files

0.0.47

2 files

0.0.46

2 files

0.0.45

2 files

0.0.44

2 files

0.0.43

2 files

0.0.42

2 files

0.0.41

2 files

0.0.40

2 files

0.0.39

2 files

0.0.38

2 files

0.0.37

2 files

0.0.36

2 files

0.0.35

2 files

0.0.34

2 files

0.0.33

2 files

0.0.32

2 files

0.0.31

2 files

0.0.30

2 files

0.0.29

2 files

0.0.28

2 files

0.0.27

2 files

0.0.26

2 files

0.0.25

2 files

0.0.24

2 files

0.0.23

2 files

0.0.22

2 files

0.0.21

2 files

0.0.20

2 files

0.0.19

2 files

0.0.18

2 files

0.0.17

2 files

0.0.16

2 files

0.0.15

2 files

0.0.14

2 files

0.0.13

2 files

0.0.12

2 files

0.0.11

2 files

0.0.10

2 files

0.0.9

2 files

0.0.8

2 files

0.0.7

2 files

0.0.6

2 files

0.0.5

2 files

0.0.4

2 files

0.0.3

2 files

0.0.2

2 files

0.0.1

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page