localarena 0.2.0

Live multi-provider model evaluation, scoring, reports, and Elo

LocalArena

Live multi-provider model evaluation, deterministic scoring, standalone HTML reports, and Elo standings.

pip install localarena

LocalArena calls the endpoints you explicitly configure. It does not bundle recorded answers, copy public leaderboard results, download models, or fall back to another provider.

Command line

Create a JSON configuration with one or more model targets and prompt tasks, then run the complete Cartesian product:

localarena run evaluation.json \
  --output results.json \
  --report report.html

Built-in profiles:

• llamacpp — http://127.0.0.1:8080/v1
• ollama — http://127.0.0.1:11434/v1
• lmstudio — http://127.0.0.1:1234/v1
• openrouter — OPENROUTER_API_KEY
• openai — OPENAI_API_KEY
• custom — an explicit compatible base URL

localarena providers
localarena models ollama
localarena report results.json --output report.html

Config files use api_key_env and headers_env; literal API keys and sensitive literal headers are rejected. Prompt and answer content is excluded from results by default. Use --include-content only when it is safe to retain that content.

See the complete configuration example and provider/task guide.

Python API

from localarena import (
    EvaluationRunner,
    NumericMatch,
    ModelTarget,
    PromptTask,
    create_provider,
    write_html_report,
)

target = ModelTarget(
    name="local-model",
    provider=create_provider("ollama"),
    model="qwen3:0.6b",
    max_tokens=64,
    temperature=0,
)
task = PromptTask.from_text(
    "arithmetic",
    "Return only the result of 6 multiplied by 7.",
    evaluator=NumericMatch(42),
)

run = EvaluationRunner(
    [target],
    [task],
    max_concurrency=2,
).run(name="Live evaluation")

print(run.to_json())
write_html_report(run, "report.html")

Deterministic evaluators include exact, containment, regular-expression, JSON, and numeric scoring. ModelJudge supports live rubric scoring with another configured target. Generation failures and scoring failures remain individual result rows instead of aborting the run.

Elo arena

The lower-level portable arena remains available:

from localarena import Arena, Result

arena = Arena(["model-a", "model-b"])
arena.record("model-a", "model-b", Result.LEFT)

for row in arena.standings():
    print(row.rank, row.name, row.rating)

Arena and evaluation snapshots are JSON-safe. The npm package implements the same arena and evaluation run contracts.