openeval-core 0.2.1

  ██████╗ ██████╗ ███████╗███╗   ██╗███████╗██╗   ██╗██████╗ ██╗
  ██╔══██╗██╔══██╗██╔════╝████╗  ██║██╔════╝██║   ██║██╔══██╗██║
  ██║  ██║██████╔╝█████╗  ██╔██╗ ██║█████╗  ██║   ██║██████╔╝██║
  ██║  ██║██╔═══╝ ██╔══╝  ██║╚██╗██║██╔══╝  ██║   ██║██╔══██╗██║
  ██████╔╝██║     ███████╗██║ ╚████║███████╗╚██████╔╝██║  ██║███████╗
  ╚═════╝ ╚═╝     ╚══════╝╚═╝  ╚═══╝╚══════╝ ╚═════╝ ╚═╝  ╚═╝╚══════╝

📌 Table of Contents

• 💡 Executive Overview & Vision
• ⚡ Performance & Benchmark Comparison
• ⚔️ OpenEval vs. Traditional LLM-as-a-Judge
• 🏗️ System Architecture & Execution Pipeline
• 📦 Installation Matrix
• 🚀 Quickstart Tutorials (4 Adapters)
  • Tier 1: Core Engine (Zero Dependencies)
  • Tier 2: Native Fixtura Trace Integration
  • Tier 3: LangChain Run Trees
  • Tier 4: OpenAI Tool Calling Messages
• 📊 Mathematical Specification of All 7 Deterministic Metrics
• 🖥️ Command Line Interface (CLI) Guide
• ⚙️ GitHub Actions CI/CD Gating
• 🔑 Core Data Model API Reference
• 📖 Repository Structure & Proposals
• ❓ Frequently Asked Questions (FAQ)
• 📄 License

💡 Executive Overview & Vision

OpenEval is a high-speed, purely deterministic evaluation engine designed specifically for AI agent trajectories.

As AI agents transition from simple single-prompt text generators to complex autonomous tool-calling loops (interacting with file systems, databases, payment APIs, and web search), evaluating their behavior requires inspecting actual execution traces.

OpenEval bypasses prompt-based LLM judges entirely. It consumes structured execution steps (TraceStep) and evaluates exact tool calls, argument correctness, trajectory step efficiency, final state transitions, permission denial recoveries, Verified Replay divergences, and environment fingerprint staleness using 100% pure mathematical Python functions.

⚡ Performance & Benchmark Comparison

Benchmark executed on 1,000 synthetic agent evaluation cases:

⚔️ OpenEval vs. Traditional LLM-as-a-Judge

Traditional Evaluation (LLM-as-a-Judge):
[ Agent Run ] ──> [ Secondary Prompt ] ──> [ GPT-4 API Call ] ──> [ Flaky Text Verdict ($$$) ]

OpenEval Deterministic Engine:
[ Agent Trace ] ──> [ Pure Python Logic ] ──> [ Instant Math Score (1.0 / 0.0 / None) ($0) ]

🏗️ System Architecture & Execution Pipeline

+---------------------------------------------------------------------------------------------------+
|                                      INPUT TRAJECTORY INGESTION                                   |
|   +--------------------------+    +---------------------------+    +--------------------------+   |
|   |  Fixtura .trace File     |    |   LangChain Run Tree      |    |   OpenAI Messages List   |   |
|   |  (Compressed JSONL)      |    |   (Traced Agent Runs)     |    |   (ChatCompletion API)   |   |
|   +------------+-------------+    +-------------+-------------+    +------------+-------------+   |
+----------------|--------------------------------|-------------------------------|-----------------+
                 |                                |                               |
                 v                                v                               v
+---------------------------------------------------------------------------------------------------+
|                                       ADAPTER SUBSYSTEM                                           |
|   +--------------------------+    +---------------------------+    +--------------------------+   |
|   |   from_fixtura_trace()   |    |   from_langchain_run()    |    |  from_openai_messages()  |   |
|   +------------+-------------+    +-------------+-------------+    +------------+-------------+   |
+----------------|--------------------------------|-------------------------------|-----------------+
                 +--------------------------------+-------------------------------+
                                                  |
                                                  v
+---------------------------------------------------------------------------------------------------+
|                                CORE UNIFIED DATA MODEL (Pure Dataclasses)                         |
|                                                                                                   |
|   AgentTrace                                                                                      |
|   ├── task_id: str                                                                                |
|   ├── input: str | final_output: str | actual_state: dict | metadata: dict                        |
|   └── steps: list[TraceStep]                                                                      |
|       ├── step_id: int | type: "thought" | "tool_call" | "tool_result"                              |
|       ├── tool_name: str | tool_args: dict | tool_result: str                                     |
|       ├── denied: bool (Permission denial / validation error tracking)                            |
|       ├── finish_reason: str | provider: str | model: str | tokens: dict | latency_ms: float        |
|       └── divergent: bool (Verified Replay trajectory divergence marker)                          |
+-------------------------------------------------|-------------------------------------------------+
                                                  |
                                                  v
+---------------------------------------------------------------------------------------------------+
|                                 PURE DETERMINISTIC EVAL ENGINE                                    |
|                                                                                                   |
|   EvalTestCase                                                                                    |
|   ├── expected_tool_calls: list[dict] | expected_final_state: dict                                    |
|   └── max_steps: int | timeout_seconds: float                                                         |
|                                                                                                   |
|   +-------------------------------------------------------------------------------------------+   |
|   |                                  METRIC PIPELINE MODULES                                  |   |
|   |  1. ToolSelectionAccuracy (Exact tool invocation ratio)                                   |   |
|   |  2. ArgumentCorrectness   (Key-value schema match precision)                              |   |
|   |  3. StepEfficiency        (Optimal vs actual steps ratio)                                 |   |
|   |  4. GoalCompletionRate    (Final state transition accuracy)                               |   |
|   |  5. DenialRecoveryRate    (Permission denial recovery without loops)                      |   |
|   |  6. DivergenceScore       (Verified Replay trajectory step fidelity)                      |   |
|   |  7. FixtureFreshness      (Environment fingerprint staleness check)                       |   |
|   +---------------------------------------------+---------------------------------------------+   |
+-------------------------------------------------|-------------------------------------------------+
                                                  |
                                                  v
+---------------------------------------------------------------------------------------------------+
|                                   VERDICT & REPORT GENERATOR                                      |
|   +---------------------------+     +---------------------------+     +-----------------------+   |
|   | MetricResult              | --> | Suite Runner              | --> | CLI & GitHub Action   |   |
|   | (score, passed, details)  |     | (openeval.runner)         |     | (openeval.report)     |   |
|   | passed: True/False/None   |     | Excludes passed=None      |     | Pure Markdown/JSON    |   |
|   +---------------------------+     +---------------------------+     +-----------------------+   |
+---------------------------------------------------------------------------------------------------+

📦 Installation Matrix

OpenEval requires zero external framework dependencies to run its core engine:

# 1. Core Engine (Zero external dependencies — pure Python)
pip install openeval-core

# 2. With Native Fixtura Trace Support (.trace compressed JSONL)
pip install "openeval-core[fixtura]"

# 3. With LangChain Run Adapter Support
pip install "openeval-core[langchain]"

# 4. With All Optional Adapters Installed
pip install "openeval-core[fixtura,langchain]"

# 5. Local Editable Installation (From Source Checkout)
pip install -e ".[fixtura]"

[!NOTE] The PyPI distribution package name is openeval-core, while the CLI executable command is openeval.

🚀 Quickstart Tutorials (4 Adapters)

Tier 1: Core Engine (Zero Dependencies)

The core engine runs anywhere without external frameworks or SDK dependencies:

from openeval.metrics import ToolSelectionAccuracy, ArgumentCorrectness
from openeval.models import AgentTrace, EvalTestCase, TraceStep

# 1. Define expectations (what the agent was supposed to do)
test_case = EvalTestCase(
    task_id="quickstart-1",
    input="Search for weather in Tokyo",
    expected_tool_calls=[{"tool": "search", "args": {"query": "weather in Tokyo"}}],
    expected_final_state={"searched": True},
    expected_output_contains=[],
    max_steps=5,
    timeout_seconds=10.0
)

# 2. Provide actual trace (what the agent actually executed)
trace = AgentTrace(
    task_id="quickstart-1",
    input="Search for weather in Tokyo",
    steps=[
        TraceStep(
            step_id=1, 
            type="tool_call", 
            content="", 
            tool_name="search", 
            tool_args={"query": "weather in Tokyo"}, 
            tool_result="85 degrees and sunny", 
            timestamp=0.0
        )
    ],
    final_output="The weather in Tokyo is 85 degrees and sunny.",
    actual_state={"searched": True},
    metadata={}
)

# 3. Score deterministically (0.0ms execution time)
metric = ToolSelectionAccuracy()
result = metric.score(trace, test_case)

print(f"Metric:  {result.metric_name}")
print(f"Score:   {result.score} (Passed: {result.passed})")
print(f"Details: {result.details}")

Tier 2: Native Fixtura Trace Integration

Ingest zstd-compressed .trace files recorded by Fixtura. OpenEval natively parses permission denials, completion finish reasons, token counts, Verified Replay divergence markers, and fingerprint drift verdicts:

from openeval.adapters.fixtura import from_fixtura_trace
from openeval.metrics import (
    ToolSelectionAccuracy, 
    DenialRecoveryRate, 
    DivergenceScore, 
    FixtureFreshness
)

# Ingest Fixtura trace file
trace = from_fixtura_trace(
    trace_path="fixtures/checkout.trace",
    task_id="task-101",
    input_text="Execute user checkout",
    final_output="Order placed successfully",
    actual_state={"order_created": True},
    metadata={}
)

# Evaluate against test case
metrics = [
    ToolSelectionAccuracy(),
    DenialRecoveryRate(),
    DivergenceScore(),
    FixtureFreshness()
]

for m in metrics:
    res = m.score(trace, test_case)
    print(f"{m.name:25s}: Score = {res.score:.2f} | Passed = {res.passed!s:5s} | {res.details}")

Tier 3: LangChain Run Trees

Convert LangChain Run trees directly into OpenEval traces:

from openeval.adapters.langchain import from_langchain_run
from langchain_core.tracers.context import collect_runs

with collect_runs() as cb:
    agent.invoke({"input": "Search for weather"})

trace = from_langchain_run(cb.traced_runs[0])

Tier 4: OpenAI Tool Calling Messages

Convert OpenAI ChatCompletion message lists into structured AgentTrace trajectories:

from openeval.adapters.openai import from_openai_messages

messages = [
    {"role": "user", "content": "Fetch weather in Tokyo"},
    {"role": "assistant", "tool_calls": [{"id": "call_1", "function": {"name": "get_weather", "arguments": "{\"location\": \"Tokyo\"}"}}]},
    {"role": "tool", "tool_call_id": "call_1", "content": "85F sunny"}
]

trace = from_openai_messages(messages)

📊 Mathematical Specification of All 7 Deterministic Metrics

OpenEval ships out-of-the-box with seven pure, deterministic evaluation metrics.

1. Tool Selection Accuracy (ToolSelectionAccuracy)

Evaluates the ratio of expected tool calls executed by the agent. $$\text{Score} = \frac{\text{Count}(T_{\text{executed}} \cap T_{\text{expected}})}{\text{Count}(T_{\text{expected}})}$$

Denial Guard: Tool calls with step.denied == True (permission denials or validation errors) are excluded from $T_{\text{executed}}$, ensuring forbidden calls don't count as successful tool selections.

2. Argument Correctness (ArgumentCorrectness)

Evaluates the key-value argument match precision for executed tool calls. $$\text{Score} = \frac{\sum \text{matching_kv_pairs}}{\sum \text{expected_kv_pairs}}$$

3. Step Efficiency (StepEfficiency)

Evaluates step count efficiency relative to the optimal step count specified in the test case. $$\text{Score} = \min\left(1.0, \frac{N_{\text{optimal}}}{N_{\text{actual_steps_taken}}}\right)$$

4. Goal Completion Rate (GoalCompletionRate)

Evaluates the accuracy of final environment state key transitions against expected target state. $$\text{Score} = \frac{\text{Count}(\text{actual_state}[k] == \text{expected_state}[k])}{\text{Total Expected Keys}}$$

5. Denial Recovery Rate (DenialRecoveryRate)

Evaluates whether an agent that encountered a permission denial or validation error:

1. Did not retry the identical forbidden tool call (tool_name + tool_args) anywhere in the remainder of the trajectory.
2. Successfully recovered by executing an allowed tool or producing output. $$\text{Score} = \frac{\text{Count}(\text{recovered_denials})}{\text{Total Denials Encountered}}$$

Zero Denials: Returns score = 1.0, passed = True, details = "No permission denials...".

6. Divergence Score (DivergenceScore)

Evaluates trajectory agreement during Verified Replay offline comparison. $$\text{Score} = \frac{\text{Step Index of First Divergence}}{\text{Total Trajectory Steps}}$$

No Divergence: Returns score = 1.0, passed = True, details = "No Verified Replay divergence detected.".

7. Fixture Freshness (FixtureFreshness)

Evaluates whether the trace fixture's tool registry fingerprint verdict indicates a fresh spec:

• verdict == "PASS" $\rightarrow$ score = 1.0, passed = True
• verdict == "DRIFTED" $\rightarrow$ score = 0.0, passed = False (genuine drift regression)
• verdict == "UNVERIFIED" $\rightarrow$ score = 0.0, passed = False (check-drift failed / config error)
• No Fingerprint Metadata $\rightarrow$ score = 1.0, passed = None (NOT EVALUATED)

[!IMPORTANT] Disambiguated Verdicts: FixtureFreshness returns passed = None when a trace was never drift-checked. In aggregate suite reporting, metrics with passed is None are excluded from pass-rate denominators, eliminating false-positive and false-negative reporting bugs in CI gates!

🖥️ Command Line Interface (CLI) Guide

OpenEval provides a high-speed CLI binary (openeval).

# Run a single evaluation test case against a trace
openeval run --trace examples/simple_agent/trace.json --testcase examples/simple_agent/testcase.json

# Run an entire evaluation suite directory
openeval run --suite tests/ --output results/

# Generate a pure Markdown evaluation report
openeval report --input results/ --format markdown

# Generate a structured JSON summary report
openeval report --input results/ --format json

[!WARNING] The openeval report command exits with exit code 1 if any JSON result file in the input directory is malformed or corrupted, guaranteeing pipeline failures on corrupted evals.

⚙️ GitHub Actions CI/CD Integration

Gate pull requests deterministically in CI:

name: Agent Trajectory Evaluation CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  eval-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Grade Agent Execution
        uses: organization/openeval-core@v1
        with:
          suite: path/to/eval_suite_dir
          fail-under: '0.8'  # Fail CI build if average pass rate < 80%

🔑 Core Data Model API Reference

OpenEval relies on pure, strongly typed dataclasses in openeval/models.py:

TraceStep Dataclass

@dataclass
class TraceStep:
    step_id: int
    type: Literal["thought", "tool_call", "tool_result", "output"]
    content: str
    tool_name: str | None
    tool_args: dict | None
    tool_result: str | None
    timestamp: float
    error: str | None = None
    
    # Extended telemetry fields
    denied: bool = False
    finish_reason: str | None = None
    provider: str | None = None
    model: str | None = None
    tokens: dict[str, int] | None = None
    latency_ms: float | None = None
    divergent: bool = False

MetricResult Dataclass

@dataclass
class MetricResult:
    metric_name: str
    score: float
    passed: bool | None  # True (Passed), False (Failed), None (Not Applicable / Un-Evaluated)
    details: str

📖 Architecture & Proposals Directory

❓ Frequently Asked Questions (FAQ)

📄 License

This project is licensed under the MIT License.