kalytera 0.1.4

Real-time quality monitoring and failure detection for production AI agents

Kalytera

Know when your AI agent fails — before your users do.

pip install kalytera

import kalytera

kalytera.configure(api_key="kly_live_...")

kalytera.trace(
    session_id="session-001",
    step_number=1,
    step_name="classify_intent",
    input="I need to cancel my subscription",
    output="I can help with that. What's the reason?",
)

That's it. Kalytera scores every step with an LLM judge and surfaces failure patterns — what's breaking, at which workflow step, and why.

→ Live demo dashboard · Get an API key

---

Get started

1. Get a free API key (no credit card):

curl -s -X POST https://api.kalytera.dev/signup \
  -H "Content-Type: application/json" \
  -d '{"email": "[e-mail]", "name": "my-agent"}' | python3 -m json.tool

Copy the api_key from the response (looks like kly_live_...).

2. Install and trace:

import kalytera

kalytera.configure(
    api_key="kly_live_...",
    api_endpoint="https://api.kalytera.dev",
    agent_id="my-agent",
)

# Call after every step in your agent workflow
kalytera.trace(
    session_id="session-001",   # same ID groups steps of one conversation
    step_number=1,
    step_name="retrieve_policy",
    input="What is the cancellation policy?",
    output="You can cancel anytime. No refunds for partial months.",
    tool_calls=[{"name": "policy_api", "success": True, "latency_ms": 210}],
)

trace() returns in under 5ms. It never raises. If the server is unreachable, your agent keeps running.

3. Open the dashboard — quality scores appear within 30 seconds:

https://app.kalytera.dev

---

What Kalytera detects

Every step is scored across six dimensions by an LLM judge (Claude Haiku):

Dimension	Default weight	What it measures
Accuracy	25%	Contextually correct answer for the situation
Goal alignment	25%	Agent stayed on what the user actually needed
Decision quality	15%	Reasoning was sound, right tools were chosen
Completeness	15%	Request fully resolved end-to-end
Helpfulness	10%	Response had practical value the user could act on
Factuality	10%	All claims grounded and true — no hallucinations

Steps scoring below 70% are flagged. Seven failure types are detected automatically:

wrong_answer · tool_failure · goal_drift · hallucination · context_loss · incomplete · loop

Weights and pass threshold are configurable per agent in the dashboard. You can also add custom metrics (e.g. helpfulness, tone, compliance) with their own weights.

---

Calibrate the judge

Every judge has bias. Calibration measures whether Kalytera's LLM judge agrees with your own judgement.

In the Trace Viewer, label any session with 👍 passed or 👎 failed. The dashboard shows judge accuracy against your labels and flags when weights need adjustment:

Agreement	Status
≥ 90%	Calibrated ✓
80–89%	Good
< 80%	Review your scoring weights

Label 10–20 sessions to get a reliable calibration reading. The more you label, the more precisely Kalytera can tell you if the judge is drifting.

---

@watch decorator

Zero-config alternative — wraps a function and captures input, output, and latency:

@kalytera.watch
def handle_request(user_input: str) -> str:
    return your_agent_logic(user_input)

---

Self-host

For teams that want data on their own infrastructure:

git clone https://github.com/priyamathur/kalytera-server
cd kalytera-server
cp .env.example .env        # add ANTHROPIC_API_KEY and DATABASE_URL
docker compose up

API at http://localhost:8000, dashboard at http://localhost:8501.

---

Roadmap

• LLM-as-judge scoring (4 built-in dimensions)
• Custom metrics (helpfulness, tone, compliance, …)
• Golden set calibration — label sessions, measure judge accuracy
• Cross-model agreement — flag low-confidence evals by comparing two judges
• Auto-calibration — suggest weight adjustments when agreement drops below threshold
• Team review queue — route low-confidence evals to a human reviewer

---

License

MIT — [e-mail]