evalio-ai 0.1.3

Python SDK for Evalio AI - LLM A/B testing platform

Evalio Python SDK

Python SDK for Evalio AI, an LLM experimentation platform for routing traffic between model or prompt variants, reporting metrics, and collecting feedback.

Package name

Install package:

pip install evalio-ai

Import path stays:

from evalio import EvalioClient

With Anthropic support:

pip install "evalio-ai[anthropic]"

Quick Start

from evalio import EvalioClient

client = EvalioClient(
    api_key="sk-evalio-your-key",
    base_url="https://evalio-ai-production.up.railway.app",
    provider_keys={"openai": "sk-your-openai-key"},
)

response = client.run(
    experiment_id=12,
    messages=[{"role": "user", "content": "How do I reset my password?"}],
    user_id="user-123",
    idempotency_key="support-reset-user-123-msg-001",
    provider_params={"temperature": 0.2},
)

print(response.content)
print(response.variant)
print(response.request_id)
print(response.cost_usd)
print(response.latency_ms)

What the SDK does

1. Calls Evalio to get a routing decision
2. Calls the assigned provider/model
3. Reports completion metrics back to Evalio
4. Returns the provider response

Sticky assignment means the same user_id stays on the same variant after the first assignment.

Per-call provider parameters

If your app needs JSON mode or other call-specific provider options, pass them at runtime:

response = client.run(
    experiment_id=12,
    messages=[{"role": "user", "content": "Return valid JSON only"}],
    user_id="user-123",
    idempotency_key="json-user-123-msg-001",
    provider_params={
        "temperature": 0.2,
        "response_format": {"type": "json_object"},
    },
)

Runtime params are merged on top of the variant's saved extra_params.

Feedback

client.feedback(
    response.request_id,
    score=1,
    comment="Helpful answer",
    idempotency_key=f"feedback:{response.request_id}:thumbs-up",
)

Retry safety

EvalioClient.run(..., idempotency_key=...) and EvalioClient.feedback(..., idempotency_key=...) can be retried safely. The backend replays the original result instead of creating duplicates.

Supported providers

• OpenAI
• Anthropic via optional extra