entropy-agent-eval 0.1.8

Entropy-based evaluation metrics for AI agent behavior, tools, trajectories, uncertainty reduction, and robustness.

Entropy-Based Evaluation of AI Agents

entropy-agent-eval implements EEA, a toolkit for measuring agent behavior with entropy metrics:

• action entropy for action-selection uncertainty
• trajectory entropy for strategy diversity
• tool entropy for tool-use specialization
• information gain for uncertainty reduction
• entropy curves for temporal behavior
• robustness summaries across repeated runs
• a configurable Entropic Agent Score

Any agent library can integrate by converting its trace events into AgentRun records. Here is the original experimental paper: https://arxiv.org/pdf/2606.05872

Who This Is For

Use EEA when you want to compare agent behavior beyond success rate:

• framework authors who want behavioral diagnostics
• application teams evaluating agent changes before deployment
• researchers comparing ReAct, planner, tool-using, or multi-agent systems
• observability teams turning traces into evaluation metrics

Install

Requires Python 3.12 or newer.

From GitHub:

pip install git+https://github.com/olahsymbo/entropy-agent-eval.git

For local development:

poetry install --with dev

Optional plotting support:

pip install "entropy-agent-eval[plots]"

Build source and wheel distributions:

poetry build

Install a local wheel:

pip install dist/entropy_agent_eval-v0.1.1-py3-none-any.whl

Release

Package builds are handled by Poetry. To cut a release:

poetry version patch
git tag v0.1.1
git push origin main --tags

Quick Start

from entropy_agent_eval import AgentRun, EntropyEvaluator

runs = [
    AgentRun.from_mapping(
        {
            "task": "Write sorting algorithm",
            "success": True,
            "cost": 0.12,
            "trajectory": ["search", "python", "test", "answer"],
            "before": {"A": 0.4, "B": 0.3, "C": 0.2, "D": 0.1},
            "after": {"A": 0.9, "B": 0.05, "C": 0.03, "D": 0.02},
        }
    )
]

report = EntropyEvaluator().evaluate(runs)
print(report.as_dict())

CLI

eea examples/runs.json
eea examples/runs.json --per-run

The CLI accepts JSON objects with a top-level runs list, raw JSON lists, or JSONL files.

Integration Model

You do not have to export JSON logs. JSON is only one supported path.

EEA needs one thing: normalized traces as AgentRun objects. Those traces can come from live callbacks, custom wrappers, databases, observability systems, JSON/JSONL files, or benchmark harnesses.

LangChain / Google ADK / custom agent / stored trace
        ↓
AgentRun
        ↓
EntropyEvaluator
        ↓
entropy metrics + Entropic Agent Score

Data Contract

The central integration type is AgentRun:

{
  "task": "qa-001",
  "success": true,
  "cost": 0.08,
  "trajectory": ["search", "read", "answer"],
  "before": {"correct": 0.45, "distractor": 0.55},
  "after": {"correct": 0.92, "distractor": 0.08}
}

For richer logs, use explicit events:

{
  "task_id": "coding-42",
  "events": [
    {"kind": "tool", "name": "search"},
    {"kind": "tool", "name": "python"},
    {"kind": "action", "name": "answer"}
  ],
  "success": true
}

Cost

cost is user or framework supplied. It can mean USD, total tokens, token-normalized cost, tool-call cost, compute cost, or any other numeric penalty you want to apply consistently across compared runs.

The evaluator reports it as mean_cost and subtracts it inside EntropicAgentScore. If cost is unknown or irrelevant, omit it or leave it as 0.0.

Full guide: docs/concepts/cost.md

Custom Agent Integration

from entropy_agent_eval import EntropyEvaluator
from entropy_agent_eval.adapters import EventRecorder

recorder = EventRecorder(task_id="task-123")
recorder.tool("search")
recorder.tool("python")
recorder.action("answer")

run = recorder.to_run(success=True, cost=0.04)
print(EntropyEvaluator().evaluate([run]).as_dict())

Full guide: docs/integrations/custom-agents.md

LangChain Integration

from entropy_agent_eval.adapters.langchain import EntropyCallbackHandler

handler = EntropyCallbackHandler(task_id="lc-001")

# Pass `handler` in your LangChain config/callbacks.
# result = chain.invoke(inputs, config={"callbacks": [handler]})

run = handler.to_run(success=True, cost=0.10)

Full guide: docs/integrations/langchain.md

Google ADK-Style Event Integration

from entropy_agent_eval.adapters.google_adk import runs_from_adk_events

run = runs_from_adk_events(
    "adk-001",
    [
        {"event_type": "tool", "tool_name": "Search"},
        {"event_type": "model", "model": "gemini"},
    ],
    success=True,
)

Full guide: docs/integrations/google-adk.md

Stored Trace Integration

If your traces are already in a database, warehouse, or observability platform, export or query them into AgentRun-compatible dictionaries and evaluate them offline.

Full guide: docs/integrations/observability.md

Metric Notes

High entropy is not automatically good. EEA treats entropy as a behavioral signature:

• low action entropy can mean focus or brittle determinism
• medium entropy can indicate adaptive branching
• high entropy can indicate exploration or chaos
• successful agents should often reduce state entropy over time
• robust agents can have moderate trajectory entropy with low outcome entropy

EntropicAgentScore is configurable:

from entropy_agent_eval import EntropicAgentScore, EntropyEvaluator

evaluator = EntropyEvaluator(
    EntropicAgentScore(
        success_weight=2.0,
        information_gain_weight=1.0,
        exploration_efficiency_weight=0.5,
        cost_weight=1.5,
    )
)

Concept guides:

• docs/concepts/overview.md
• docs/concepts/metrics.md
• docs/concepts/cost.md

Benchmark

Any callable that accepts a BenchmarkTask and returns an AgentRun or compatible dictionary can be benchmarked:

from entropy_agent_eval.benchmarks import QA_TASKS, run_benchmark

def agent(task):
    return {
        "task_id": task.id,
        "trajectory": ["think", "answer"],
        "success": True,
    }

runs = run_benchmark(QA_TASKS, agent)

Controlled Benchmark

The experiments directory contains a controlled benchmark that compares reference agent patterns across factual QA, multi-hop, and coding tasks.

poetry run python scripts/run_experiment.py

The script writes normalized runs and per-agent summaries to experiments/results/.

Learning Roadmap Agent Experiment

The project also includes a framework-backed experiment for a Learning Roadmap Agent. It can run with LangChain, Google ADK, or both when the optional dependencies and API keys are installed.

pip install "entropy-agent-eval[langchain]"
export OPENAI_API_KEY="..."
poetry run python scripts/run_learning_roadmap_experiment.py --provider langchain

pip install "entropy-agent-eval[google-adk]"
export GOOGLE_API_KEY="..."
poetry run python scripts/run_learning_roadmap_experiment.py --provider google-adk

The roadmap experiment runner also reads .env automatically. For Google ADK, set GOOGLE_API_KEY or GEMINI_API_KEY.

Full guide: docs/experiments/learning-roadmap-agent.md

Contributing

See CONTRIBUTING.md. New adapters are welcome, especially for frameworks that can expose tool calls, model calls, actions, costs, outcomes, and uncertainty states.

License

MIT. See LICENSE.