kalibra 0.2.4

Kalibra
The diff tool for AI agent runs.
The CLI that catches what the dashboard misses.

Success rate: 80% → 80%. Duration: flat. Tokens: flat. Everything looks the same — but 2 task types that always passed started failing, and 2 that always failed started passing. The aggregate hid it. The per-task breakdown caught it.

"Unsuccessful AI products almost always share a common root cause: a failure to create robust evaluation systems." — Hamel Husain, Your AI Product Needs Evals

pip install kalibra
kalibra compare baseline.jsonl current.jsonl -v
kalibra demo    # try it with sample data

Who this is for

• Teams running agent evals in CI who want a regression gate without adopting a dashboard
• Anyone who's been burned by averages hiding regressions
• Developers who prefer a CLI and a config file over another UI to log into

What it doesn't do

• Not a tracing backend. It reads Phoenix, OTel GenAI, Langfuse, and flat JSONL exports.
• Not a dashboard. Output is terminal text, markdown, or JSON.
• Not an LLM judge. No model calls, no API keys, no evaluator prompts.
• Doesn't replace Phoenix or Langfuse. It compares the traces they produce.

What it does

• Statistically transparent — two-proportion z-test on rates, percentile bootstrap (n=1000) on continuous metrics. Every number has a named method behind it.
• Significance-gated thresholds — success_rate_delta >= -2 fails your CI pipeline (exit 1) only when the change is statistically significant; insignificant deltas skip the gate instead of failing it
• Per-task and per-span breakdown — catches regressions that cancel out in the aggregate
• Two dependencies — click + pyyaml. No ML frameworks, no API keys, no LLM calls

Quality gates for CI

# kalibra.yml
baseline:
  path: ./baselines/production.jsonl
current:
  path: ./eval-output/canary.jsonl

require:
  - success_rate_delta >= -2     # max 2pp success rate drop
  - regressions <= 5             # max 5 tasks regressed
  - cost_delta_pct <= 20         # max 20% cost increase

kalibra compare        # reads kalibra.yml, exits 1 on failure

GitHub Actions

- uses: khan5v/kalibra-action@v1
  with:
    baseline: baselines/production.jsonl
    current: current.jsonl
    config: kalibra.yml

Posts a markdown report as a PR comment. Exits 1 on gate failure.

name: Agent Quality Gate
on: [pull_request]

jobs:
  kalibra:
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
    steps:
      - uses: actions/checkout@v5
      - run: python eval.py --output current.jsonl
      - uses: khan5v/kalibra-action@v1
        with:
          baseline: baselines/production.jsonl
          current: current.jsonl
          config: kalibra.yml

Integrations

Kalibra auto-detects trace formats. Each tutorial works without an API key.

Integration	Trace format	Demo scenario	Tutorial
Phoenix / OpenInference	llm.*, openinference.*	Multi-step agent with span tree aggregation
OTel GenAI	gen_ai.*	Truncation regression hidden by aggregate improvement
CrewAI	Flat JSONL	Failure redistribution and cost explosion

sources:
  baseline:
    path: ./traces.jsonl
    where:
      - variant == baseline
  current:
    path: ./traces.jsonl
    where:
      - variant == current

fields:
  outcome: metadata.result
  cost: agent_cost.total_cost
  task_id: metadata.task_name

kalibra compare a.jsonl b.jsonl --outcome metadata.result --cost usage.total_cost

baseline:
  path: ./langfuse.jsonl
  fields: { outcome: metadata.result, cost: usage.total_cost }
current:
  path: ./braintrust.jsonl
  fields: { outcome: scores.correctness, cost: metrics.cost }

from kalibra.loader import load_traces
from kalibra.engine import compare
from kalibra.renderers import render

baseline = load_traces("baseline.jsonl")
current = load_traces("current.jsonl")

result = compare(baseline, current, require=["success_rate_delta >= -5"])
print(render(result, "terminal", verbose=True))
print("passed:", result.passed)

Development

git clone https://github.com/khan5v/kalibra.git
cd kalibra
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest