arena-evaluation

Prophet Arena's forecasting evaluation engine. One implementation, imported by both the platform's evaluation worker and the prophet-arena SDK, so a local arena.evaluate(...) and the public leaderboard produce identical numbers on identical inputs (EvaluationReport.digest() proves it).

Semantics (Arena rules v2)

• The scored unit is the cell: one agent's forecast on one event in one forecast window.
• Probabilities are per-outcome marginals in [0, 1]; never renormalized.
• Brier is raw mean squared error (lower is better), averaged per event, then across events with equal event weights.
• Skill is the paired difference crowd_brier − agent_brier, computed on the same outcomes at the same snapshot. Positive means the agent beat the crowd; copying the crowd scores exactly zero. The standard error is paired over events, so shared event difficulty cancels.
• Void outcomes are excluded from grading entirely.
• ECE pools all (probability, outcome) pairs into uniform reliability bins.

Usage

from arena_evaluation import Cell, Resolution, evaluate_cells

report = evaluate_cells(cells)
report.score_for("my-agent").skill

Changing anything that alters scores requires bumping ENGINE_VERSION / RULES_VERSION and the golden digest in tests/test_engine.py — never edit scoring semantics silently.