exgentic 0.3.0

Exgentic - General agent evaluation

Evaluate any agent on any benchmark in the simplest way possible

---

What is Exgentic?

Exgentic is a universal evaluation framework that enables standardized testing of AI agents across diverse benchmarks and domains. It provides a consistent interface for evaluating any agent on any benchmark, making it easy to compare performance, reproduce results, and ensure your agent works reliably across different tasks and environments.

Who is it for?

1. General Audience - Visit www.exgentic.ai to explore the first general agent leaderboard comparing leading agents and frontier models across varied tasks.
2. Agent Builders - Evaluate your agents comprehensively across multiple domains and benchmarks.
3. Researchers & Component Developers - Test agentic components (memory, context compression, planning) across different agents and domains.
4. Benchmark Builders - Evaluate your benchmark across multiple agents to ensure meaningful differentiation.

---

Quick Start

Installation

uv tool install exgentic

API Credentials

export OPENAI_API_KEY=...
# or
export ANTHROPIC_API_KEY=...

Run an Evaluation

# List available benchmarks and agents
exgentic list benchmarks
exgentic list agents

# Evaluate an agent on a benchmark
exgentic evaluate --benchmark tau2 --agent tool_calling --subset retail --num-tasks 2 \
  --model gpt-4o \
  --set benchmark.user_simulator_model="gpt-4o"

Benchmarks are automatically set up in an isolated virtual environment on first run — no manual installation needed. You can also set them up explicitly:

exgentic setup --benchmark tau2
exgentic setup --agent litellm_tool_calling

For full container isolation, use the Docker runner (--set benchmark.runner=docker). You only need Docker installed and running:

exgentic evaluate --benchmark tau2 --agent tool_calling --subset retail --num-tasks 2 \
  --model gpt-4o \
  --set benchmark.runner=docker \
  --set benchmark.user_simulator_model="gpt-4o"

Python API

To use exgentic as a library, install it first:

uv add exgentic   # or: pip install exgentic

from exgentic import evaluate

results = evaluate(
    benchmark="tau2",
    agent="tool_calling",
    subset="retail",
    num_tasks=2,
    model="gpt-4o",
    benchmark_kwargs={"user_simulator_model": "gpt-4o"},
)

For more examples, see the examples/ directory.

---

Available Benchmarks

exgentic list benchmarks

Benchmark	Description
tau2	Simulated customer support tasks across multiple domains (retail, airline, banking)
appworld	Multi-app API environment testing agents' ability to interact with application interfaces
browsecompplus	Web search and browsing benchmark for information retrieval and navigation
swebench	Software engineering benchmark for resolving real-world GitHub issues
hotpotqa	Multi-hop question answering over Wikipedia
gsm8k	Grade school math word problems with optional calculator tool

Available Agents

Agent	Description
LiteLLM Tool Calling	Generic tool-calling agent via LiteLLM
SmolAgents	HuggingFace SmolAgents framework
OpenAI MCP	OpenAI Responses API with MCP tools
Claude Code	Anthropic Claude Code agent
Codex CLI	OpenAI Codex CLI agent
Gemini CLI	Google Gemini CLI agent

---

Dashboard

exgentic dashboard

---

Output Structure

Each run creates its own directory under outputs/<run_id>/:

outputs/<run_id>/
├── results.json                    # Overall scores, costs, per-session statistics
├── benchmark_results.json          # Benchmark-specific aggregated results
├── run/
│   ├── config.json                # Snapshot of benchmark and agent configuration
│   ├── run.log                    # Main execution log
│   └── warnings.log               # Warnings during execution
└── sessions/<session_id>/
    ├── config.json                # Session configuration
    ├── results.json               # Session results
    ├── trajectory.jsonl           # One JSON line per step (action + observation)
    ├── agent/
    │   └── agent.log             # Agent execution log
    └── benchmark/
        ├── results.json          # Benchmark-specific results
        └── session.log           # Benchmark session log

---

CLI Reference

# Discover
exgentic list benchmarks
exgentic list subsets --benchmark tau2
exgentic list tasks --benchmark tau2 --subset retail --limit 5
exgentic list agents
exgentic setup --benchmark tau2

# Run
exgentic evaluate --benchmark tau2 --agent tool_calling --subset airline --num-tasks 10
exgentic batch run --benchmark tau2 --agent tool_calling --subset airline --num-tasks 10

# Inspect
exgentic status --benchmark tau2 --agent tool_calling --subset airline --num-tasks 10
exgentic preview --benchmark tau2 --agent tool_calling --subset airline --num-tasks 10
exgentic results --benchmark tau2 --agent tool_calling --subset airline --num-tasks 10

# Analyze
exgentic compare --agents tool_calling openai --benchmark tau2

# Explore
exgentic dashboard

---

Advanced

Model Configuration

exgentic evaluate --benchmark tau2 --agent tool_calling --subset retail --num-tasks 2 \
  --set agent.model.temperature=0.2

Supported fields: temperature, top_p, max_tokens, reasoning_effort, num_retries, retry_after, retry_strategy

Run Limits

exgentic evaluate --benchmark tau2 --agent tool_calling --subset retail --num-tasks 2 \
  --max-steps 100 --max-actions 100

Sessions stop at either limit and record limit_reached status. Default: 100 for both.

HuggingFace

Use HuggingFace models or run evaluations on HuggingFace Jobs. See docs/huggingface.md.

---

How It Works

To learn more about Exgentic's architecture and design, see our arXiv paper.

Development

For local development, editing, and contributing, see DEVELOPMENT.md.

Contributing

We welcome issues and pull requests! See CONTRIBUTING.md for guidelines.

Citing Exgentic

@misc{bandel2026generalagentevaluation,
      title={General Agent Evaluation},
      author={Elron Bandel and Asaf Yehudai and Lilach Eden and Yehoshua Sagron and Yotam Perlitz and Elad Venezian and Natalia Razinkov and Natan Ergas and Shlomit Shachor Ifergan and Segev Shlomov and Michal Jacovi and Leshem Choshen and Liat Ein-Dor and Yoav Katz and Michal Shmueli-Scheuer},
      year={2026},
      url={https://arxiv.org/abs/2602.22953},
}

License

Apache License 2.0 — see LICENSE.

Support

For questions and support, open an issue on GitHub.