orcheval 0.1.0

Evaluations for multi-agent orchestrations.

OrchEval

Evaluate, profile, and debug multi-agent LLM systems.

Why OrchEval

OrchEval arose out of a need to observe a multi-agent class project; I wanted to know exactly how my system was behaving; when did it reroute, when did it fail, and why? Existing tools on the market (LangSmith, Langfuse, AgentOps, what have you) require accounts or self-hosted deployments, and I'm ultimately too lazy for that. Besides, some of these services aren't free.

So anyways, I thought I'd make my own. With a simple pip install and a few imports, you can get your orchestration traced and analyzed within a few lines. OrchEval doesn't do any production monitoring or prompt management, but it is quite useful for deep offline analysis of multi-agent behavior that you couldn't see otherwise, such as:

• Why is my agent's prompt growing 120% across retries?
• Is the router oscillating between two nodes instead of converging?
• Did the architecture change actually reduce cost, or just shift it?
• Which of these 50 traces is the outlier, and why?

If you're building with LangGraph, OpenAI Agents SDK, or any other custom orchestration and you want to understand what your agents are actually doing, for free, without signing up for anything, OrchEval is your tool.

Comparison

	OrchEval	LangSmith	Langfuse	AgentOps	Arize Phoenix
Type	Python library	Cloud platform	Platform (cloud + self-host)	Cloud platform	Platform (cloud + self-host)
Setup	pip install	Account + API key	Docker or cloud account	Account + API key	Docker or cloud account
Pricing	Free (Apache 2.0)	Free tier, then $39/seat/mo + per-trace	Free tier, self-host free	Free tier, then $40/mo	Free tier, self-host free (ELv2)
Data stays local	Always	No (cloud)	Self-host only	No (cloud)	Self-host only
Multi-agent routing analysis	Built-in (pattern detection)	Manual trace inspection	Manual trace inspection	Session replay	Manual trace inspection
Cross-run aggregation	Built-in (outliers, trends, shapes)	Dashboard filtering	Dashboard filtering	Dashboard filtering	Dataset comparison
Convergence tracking	Built-in (per-metric classification)	—	—	—	—
LLM behavioral patterns	Built-in (5 detectors)	—	—	—	—
Run-to-run diff	Structured (compare_runs())	Side-by-side in UI	—	—	Experiment comparison
Output format	Pydantic models, HTML, JSON, Mermaid, DataFrame	Web dashboard	Web dashboard	Web dashboard	Web dashboard + notebooks
CI/CD friendly	Yes (pure functions, no I/O)	Via API	Via API	Via API	Via API
Framework lock-in	None	Best with LangChain	None	None	None

What OrchEval Analyzes

• Cost and token breakdown by node and model
• Routing decision audit with pattern detection (invariant routing, oscillation, dominant paths)
• Multi-pass convergence tracking with per-metric trend classification
• Retry and error pattern analysis with success rates
• LLM behavioral patterns (prompt growth, stuck agents, redundant tool calls)
• Execution timeline with span hierarchy and state diffs
• Cross-run aggregation with outlier detection and trend analysis

Architecture

OrchEval has three layers:

Collect         →    Report          →    Inspect
─────────────        ──────────────       ──────────────
Adapters emit        report() runs        to_digest()
framework events     6 analysis modules   to_html()
into Traces          compare_runs()       to_mermaid()
                     TraceCollection      to_dataframe()

For contributor details, see the directory READMEs: src/orcheval/ | src/orcheval/adapters/ | src/orcheval/report/ | tests/

Installation

pip install orcheval                    # core (pydantic only)
pip install orcheval[langgraph]         # + LangGraph adapter
pip install orcheval[openai_agents]     # + OpenAI Agents SDK adapter
pip install orcheval[pandas]            # + DataFrame export

Quickstart

from orcheval import Tracer

tracer = Tracer(adapter="langgraph")
result = graph.invoke(input, config={"callbacks": [tracer.handler]})
trace = tracer.collect()

print(trace.node_sequence())   # ['planner', 'coder', 'reviewer']
print(trace.total_cost())      # 0.025
print(trace.total_tokens())    # {'prompt': 950, 'completion': 350, 'total': 1300}

# Compact text digest — feed it to an LLM for analysis
print(trace.to_digest())

# Trace Digest: a1b2c3d4

## Overview
- **Nodes:** planner → coder → reviewer (3 unique)
- **Duration:** 5300ms
- **Cost:** $0.025 (950 prompt + 350 completion = 1300 tokens)
- **Errors:** 0

## Execution Flow
1. **planner** — 1 LLM call (gpt-4o), 1500ms
2. **coder** — 1 LLM call (gpt-4o), 1 tool call (execute_code), 3000ms
3. **reviewer** — 1 LLM call (gpt-4o-mini), 800ms

Reports

from orcheval import report

full = report(trace)

# Cost breakdown
full.cost.total_cost            # 0.025
full.cost.most_expensive_node   # 'coder'
full.cost.most_expensive_model  # 'gpt-4o'

# Routing audit
full.routing.total_decisions    # 2
full.routing.flags              # [RoutingFlag(flag_type='invariant_routing', ...)]

# Convergence (for multi-pass systems)
full.convergence.is_converging  # True
full.convergence.total_passes   # 3

# Timeline
full.timeline.total_duration_ms # 5300.0

# Retries and errors
full.retries.total_errors       # 0
full.retries.retry_sequences    # []

# LLM behavioral patterns
full.llm_patterns.patterns      # [LLMPattern(pattern_type='prompt_growth', ...)]

Individual reports can be generated separately:

from orcheval.report import cost_report, routing_report

cost = cost_report(trace)
routing = routing_report(trace)

Routing Flags

OrchEval detects suspicious routing patterns automatically:

for flag in full.routing.flags:
    print(f"[{flag.flag_type}] {flag.description}")
# [invariant_routing] planner always routes to coder (3/3 decisions)
# [dominant_path] coder routes to reviewer 95%+ of the time

Flag types: invariant_routing, context_divergence, dominant_path, oscillation.

LLM Patterns

for p in full.llm_patterns.patterns:
    print(f"[{p.severity}] {p.pattern_type}: {p.description}")
# [warning] prompt_growth: coder input tokens grew 120% across invocations
# [warning] redundant_tool_call: execute_code called 3x with identical input
# [info] output_not_utilized: reviewer LLM output produced but state unchanged

Pattern types: prompt_growth, repeated_output, redundant_tool_call, system_message_variance, output_not_utilized.

HTML Visualization

trace.to_html("trace.html")  # writes to orcheval_outputs/trace.html

Generates a self-contained HTML file (no external dependencies) with:

• Summary metrics panel (duration, cost, tokens, errors)
• Interactive waterfall timeline with swimlane layout per node
• Click-to-expand detail panels showing LLM calls, tool calls, errors, and state diffs

All export methods that accept a bare filename (e.g. "trace.html") write to the orcheval_outputs/ directory by default. Paths with a directory component or absolute paths are used as-is.

Open orcheval_outputs/trace.html in any browser.

https://github.com/user-attachments/assets/04f69cc1-c911-4259-b108-fc323852031c

Text Digest

# Compact overview
print(trace.to_digest())

# Focus on specific nodes, collapse others into a summary line
print(trace.to_digest(focus_nodes=["coder"]))

# Include full LLM prompt/response content
print(trace.to_digest(include_llm_content=True))

# Control output size (~4 chars per token)
print(trace.to_digest(max_chars=8_000))

# Reuse a precomputed FullReport
print(trace.to_digest(reports=full))

Run Comparison

from orcheval import compare_runs

diff = compare_runs(baseline_trace, experiment_trace)

# Natural-language summary of all changes
print(diff.summary)
# "Cost increased $0.025 → $0.031 (+24.0%). coder duration flagged: 3000ms → 4200ms (+40.0%)."

# Programmatic access
diff.cost.total_delta.delta       # 0.006
diff.cost.total_delta.pct_change  # 24.0
diff.duration.total_delta.flagged # True
diff.errors.new_errors            # []
diff.llm_patterns.new_patterns    # [PatternDiff(...)]

# Or compare directly from a trace
diff = baseline_trace.compare(experiment_trace)

Cross-Run Aggregation

from orcheval import TraceCollection

collection = TraceCollection.from_traces(trace1, trace2, trace3, trace4, trace5)
# Or load from a directory of JSON files:
# collection = TraceCollection.from_json_dir("traces/")

# Aggregate statistics
summary = collection.summary()
summary.trace_count                # 5
summary.total_cost.mean            # 0.027
summary.total_cost.p95             # 0.035
summary.unique_node_names          # ['planner', 'coder', 'reviewer']

# Per-node breakdown
coder_stats = collection.node_stats("coder")
coder_stats.duration.median        # 3100.0
coder_stats.cost.mean              # 0.015
coder_stats.error_rate             # 0.2

# Outlier detection
outliers = collection.find_outliers("cost", threshold=2.0)
for o in outliers:
    print(f"Trace {o.trace_id}: {o.metric}={o.value:.3f} (median={o.median:.3f}) — {o.reason}")
# Trace abc123: cost=0.052 (median=0.027) — cost is 1.93x the median (threshold: 2.0x)

# Execution shape clustering
for shape in collection.execution_shapes():
    print(f"{shape.node_sequence} — {shape.trace_count} traces ({shape.fraction:.0%})")
# ['planner', 'coder', 'reviewer'] — 4 traces (80%)
# ['planner', 'coder', 'coder', 'reviewer'] — 1 trace (20%)

# Trend analysis
trend = collection.trend("cost")
trend.direction   # 'increasing'
trend.change_pct  # 15.2

Mermaid Export

print(trace.to_mermaid())

graph LR
    planner["planner (1x)"]
    coder["coder (1x)"]
    reviewer["reviewer (1x)"]
    planner -->|"1x"| coder
    coder -->|"1x"| reviewer

GitHub renders Mermaid blocks natively. Nodes with errors are highlighted in red.

DataFrame Export

df = trace.to_dataframe()  # requires pip install orcheval[pandas]

One row per event. Columns include: event_type, timestamp, node_name, span_id, duration_ms, model, input_tokens, output_tokens, cost, tool_name, error_type, source_node, target_node, and more.

Framework Support

LangGraph

from orcheval import Tracer

tracer = Tracer(adapter="langgraph")
result = graph.invoke(input, config={"callbacks": [tracer.handler]})
trace = tracer.collect()

Options:

# Infer routing decisions from node transitions
tracer = Tracer(adapter="langgraph", infer_routing=True)

# Capture input/output state on each node
tracer = Tracer(adapter="langgraph", capture_state=True)

OpenAI Agents SDK

Note: The OpenAI Agents SDK adapter is experimental and has not been validated against real workloads. If you encounter issues, please open an issue or use the ManualAdapter as a fallback.

from orcheval import Tracer
from agents.tracing import add_trace_processor

tracer = Tracer(adapter="openai_agents")
add_trace_processor(tracer.handler)

result = await Runner.run(agent, "Summarize the document")
trace = tracer.collect()

Options:

# Infer routing decisions between agents
tracer = Tracer(adapter="openai_agents", infer_routing=True)

# Capture agent metadata (name, tools, handoffs, output_type)
tracer = Tracer(adapter="openai_agents", capture_state=True)

Manual Adapter

For frameworks without a built-in adapter:

from orcheval import Tracer

tracer = Tracer()  # defaults to manual
a = tracer.adapter

a.node_entry("agent")
a.llm_call(node_name="agent", model="gpt-4o", input_tokens=150, output_tokens=80, cost=0.005)
a.tool_call("search", node_name="agent", tool_input={"query": "test"}, tool_output="result")
a.node_exit("agent", duration_ms=3000.0)

trace = tracer.collect()

Same Trace object, same analysis, regardless of framework.

Saving and Loading Traces

# Serialize
json_str = trace.to_json()          # returns JSON string
trace.to_json("trace.json")         # also writes to orcheval_outputs/trace.json
d = trace.to_dict()

# Deserialize
from orcheval import Trace
loaded = Trace.from_json(json_str)          # from string
loaded = Trace.from_json_file("trace.json") # from file
loaded = Trace.from_dict(d)

# Merge multiple traces
combined = Trace.merge(trace1, trace2, trace3)

Generate HTML from saved files

from orcheval import html_from_files

# From trace file only (report auto-generated)
html_from_files("orcheval_outputs/trace.json", output_path="trace.html")

# From trace + pre-computed report
html_from_files("orcheval_outputs/trace.json", "report.json", output_path="trace.html")

# Or load individually
from orcheval import Trace, FullReport
trace = Trace.from_json_file("orcheval_outputs/trace.json")
report = FullReport.from_json_file("report.json")
trace.to_html("trace.html", reports=report)

Contributing

Contributions are welcome. OrchEval follows a standard open-source workflow:

1. Open an issue first. Whether it's a bug report, feature request, or a question — start with a GitHub issue. This lets us discuss the approach before you write code.

2. Fork and branch. Fork the repo, create a branch from main, and make your changes. Keep commits focused — one logical change per PR.

3. Follow existing conventions. The codebase uses ruff for linting and formatting, mypy for type checking, and pytest for tests. Run all three before submitting:

   pip install -e ".[dev]"
   ruff check src/ tests/
   mypy src/
   pytest
   

4. Write tests. If you're adding a feature, add tests. If you're fixing a bug, add a test that reproduces it. Use the ManualAdapter and _ts() helpers from tests/conftest.py — no framework dependencies or API keys needed.

5. Submit a PR. Reference the issue number, describe what changed and why, and keep the diff reviewable. Large changes should be discussed in the issue first.

Adding a new adapter

See src/orcheval/adapters/README.md for the full guide. The short version: subclass BaseAdapter, implement get_callback_handler(), register in Tracer._resolve_adapter(), and add an optional-dependency group in pyproject.toml.

Adding a new report module

See src/orcheval/report/README.md. Write a pure function your_report(trace: Trace) -> YourReport with frozen Pydantic output models, add it to FullReport, and export from __init__.py.

Known Limitations

• LangGraph routing detection: LangGraph does not provide explicit routing/conditional-edge callbacks. Pass infer_routing=True to Tracer to emit inferred RoutingDecision events based on node transition sequences. These carry metadata={"inferred": True} and may not reflect actual conditional logic. For precise routing data, use the manual adapter's routing_decision() method.

• Pass boundaries / convergence tracking: Neither the LangGraph adapter nor the OpenAI Agents adapter emits PassBoundary events automatically. convergence_report() will always return an empty report unless you record pass boundaries manually via ManualAdapter.pass_boundary(). This means convergence analysis requires explicit user instrumentation regardless of framework.

• Cost data: LLMCall.cost is a passthrough field — it is auto-populated when the LLM provider reports cost in the callback response. OrchEval does not include a built-in pricing table. When cost is unavailable, the cost report falls back to token counts and call counts.

License

Apache 2.0