evalora 0.1.0

Easy AI evals — drop into any AI code in 4 lines

evalora

Dead-simple LLM evals. No account needed to start.

Install

pip install evalora

Quick Start

from evalora import Evalora

ev = Evalora()
run = ev.run()
run.check(lambda p: "4", "What is 2+2?", expected="4")
run.done()

No account, no API key. Install, run, see results.

Key Features

• Local-first -- works offline with zero config, results save to .evalora/
• Exact match + semantic similarity -- normalized string matching or embedding-based cosine similarity
• AI judge panels -- 10 built-in presets, binary or rubric scoring with majority vote
• Templates -- 7 pre-configured eval setups: rag-quality, chatbot, code-gen, summarization, translation, safety-audit, classification
• Built-in validators -- max_length, no_refusal, is_json, and more -- pass as strings or imports
• Ghost mode -- polymorphic: True, "light", "hash", or per-field dict control
• Custom scorers -- functions returning 0.0-1.0 with configurable threshold
• Auto-logging -- ev.wrap(OpenAI()) captures all LLM calls
• Framework integrations -- LangChain, LlamaIndex, CrewAI, OpenAI Agents
• Cost guard -- max_cost=5.00 stops eval early at 90% budget
• Webhooks -- POST results on completion, auto Slack format
• Trend detection -- ev.trend("name", last=10, fail_on_regression=True)
• File loading -- eval_each("data.csv") or .json or .jsonl
• Async support -- async functions auto-detected in eval_each
• Resume on crash -- checkpoint after every item, pick up where you left off
• Pairwise comparison -- head-to-head model comparison with position bias randomization
• pytest plugin -- evalora_run fixture
• Connected mode -- send results to evalora.dev for dashboards and team collaboration

Modes at a Glance

Mode	How it works	Config
Exact match	Normalized string matching (negation-aware)	scoring="exact" (default)
Semantic similarity	Cosine similarity via embeddings	embedding_api_key="sk-..." (auto-detected)
AI judges (binary)	LLM votes pass/fail with reason	judges=["Accuracy", "Safety"]
AI judges (rubric)	LLM scores 1-5 on criteria	judges=[{..., "criteria": [...]}]
Templates	Pre-configured judge + validator combos	template="rag-quality"
Custom scorers	Your functions return 0.0-1.0	scorers=[fn]
Validators	Instant checks before judges	validators=["not_empty", "is_json"]
Pairwise	Head-to-head comparison of two runs	run_pairwise(...)
Human review	Items go to dashboard review queue	Dashboard config

Examples

Basic eval

ev = Evalora()
run = ev.run(name="math test")
run.eval_each([
    ("What is 2+2?", "4"),
    ("What is 3+1?", "4"),
], fn=ask)

AI judges (string presets)

run = ev.run(
    judges=["Accuracy", "Safety", "Helpfulness"],
    provider_keys={"openai": "sk-..."},
)
run.eval_each(items, fn=ask)

Template

run = ev.run(
    template="rag-quality",
    provider_keys={"openai": "sk-..."},
)
run.eval_each("questions.csv", fn=ask)

Semantic similarity

run = ev.run(
    embedding_api_key="sk-...",  # auto-detects semantic mode
    similarity_threshold=0.85,
)
run.check(ask, "What is the capital of France?", expected="Paris is the capital")
run.done()

Ghost mode

run = ev.run(ghost="full")       # redact everything
run = ev.run(ghost="light")      # redact responses only
run = ev.run(ghost="hash")       # hash all text
run = ev.run(ghost=True, ghost_server=False)  # fully offline

Auto-log with wrap

from openai import OpenAI

client = ev.wrap(OpenAI(), run=run)
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "What is 2+2?"}],
)
# ^ auto-logged to the run

CI pipeline

ev = Evalora(api_key="evk_...", verbose=False)
run = ev.run(eval_id="...", name="nightly regression")
status = run.eval_each(fn=ask, raise_on_fail=True)
# Raises EvaloraError if any items fail

Connected Mode

Sign up at evalora.dev, grab an API key, and results appear in your dashboard:

ev = Evalora(api_key="evk_...")
run = ev.run(eval_id="your-eval-id", name="production check")
run.eval_each(fn=ask)

The dashboard is the source of truth for scoring mode, server judges, and human review. See Getting Started for details.

Documentation

• Getting Started -- install, first eval, local vs connected mode
• Scoring Modes -- exact match, semantic similarity, judges, human review
• AI Judges -- presets, binary, rubric, majority voting, error handling
• Validators -- built-in library, string shorthand, custom validators
• Configuration -- all parameters for Evalora(), run(), wrap(), trend(), and eval_each()
• Advanced -- ghost mode, templates, scorers, wrap, webhooks, file loading, async, integrations, pytest, trend, resume, pairwise
• API Reference -- classes, methods, data types, exceptions

Dashboard

evalora.dev -- run history, comparisons, team collaboration, and webhook notifications.

License

MIT