palace-eval 1.0.0

A Platform for Automated LLMs Agentic Capabilities Evaluation.

PALACE

A platform for evaluating LLM capabilities across diverse benchmarks, with native support for agentic evaluation.

Overview

PALACE (Platform for Automated LLMs Agentic Capabilities Evaluation) is an open-source evaluation framework developed by the European Commission Joint Research Centre.

Key features:

• 27+ vetted benchmarks covering reasoning, knowledge, safety, multilingual, multimodal, and agentic capabilities
• Standardized format - portable JSON-based tasklist format, distributed via HuggingFace
• Agentic-native — first-class support for tool-using agents via Vivarium runtime
• Any OpenAI-compatible endpoint - evaluate models from OpenAI, Anthropic, Mistral, local deployments, or any compatible API

Installation

From PyPI (recommended)

pip install palace-eval

Or with uv (faster):

uv pip install palace-eval

From source

git clone https://github.com/palace-ai/palace-eval.git
cd palace-eval
uv sync        # recommended
# or: pip install -e .

Requirements

• Python 3.13+
• An OpenAI-compatible API endpoint and key

Agentic evaluation (optional)

For agentic benchmarks (GAIA, AssistantBench, etc.), you also need:

• Docker 24+
• The Vivarium SDK

Note: Vivarium is currently pending open-source release (expected within weeks). The GitHub link above is not yet active. If you need agentic evaluation now, please contact [e-mail] for early access.

Install Vivarium:

pip install vivarium-ai

Vivarium starts automatically when you run an agentic tasklist.

Quick Start

# 1. Download a benchmark tasklist
palace-download -t SimpleQA

# 2. Run evaluation
palace-run -u https://api.openai.com/v1 -k $OPENAI_API_KEY -m gpt-4o -t SimpleQA -l 20

Results are saved to ~/.cache/palace/results/.

Usage

Interactive CLI

palace-cli

Guides you through selecting endpoint, model, benchmarks, and run configuration.

Direct command

palace-run \
   -u https://api.openai.com/v1 \
   -k $OPENAI_API_KEY \
   -m gpt-4o \
   -t SimpleQA \
   -l 50

See palace-run --help for all options.

Programmatic API

from palace import evaluate

evaluate(
    run_name="my-evaluation",
    url="https://api.openai.com/v1",
    token="sk-...",
    name="gpt-4o",
    tasklist="SimpleQA",
    limit=100,
)

Supported Benchmarks

PALACE includes 27+ benchmarks across multiple categories:

Category	Benchmarks
Knowledge & Reasoning	SimpleQA, HotpotQA, Humanity's Last Exam, GPQA Diamond, MUSR
Academic & Math	MMLU, MMLU-Pro, MATH-500, AIME 2025, BBH, HellaSwag
Multilingual	MMMLU (14 languages), MGSM, Belebele (122 languages)
Long Context	BABILong (4k-128k), LongBench v2
Multimodal	MMMU, MMMU Pro, VLSBench
Instruction Following	IFEval
Agentic	GAIA, AssistantBench

Download benchmarks with:

palace-download                    # all benchmarks
palace-download -t SimpleQA MMLU   # specific benchmarks

Some benchmarks (marked with a lock icon in the docs) require a HuggingFace token for gated datasets.

Tasklist Format

PALACE uses a standardized, portable JSON format for benchmarks:

<tasklist_name>/
+-- info.json      # metadata: task type, category, modalities
+-- tasks.json     # list of evaluation tasks
+-- task_files/    # optional: attachments (images, documents)

Create custom benchmarks by adding folders to ~/.cache/palace/tasklists/.

Documentation

Full documentation: https://palace.pages.code.europa.eu/palace-eval

Contributing

We welcome contributions! See CONTRIBUTING.md for guidelines.

License

Copyright (C) 2025 European Union

Licensed under the European Union Public Licence (EUPL) v. 1.2.

See LICENSE and NOTICE for details.

Citation

If you use PALACE in your research, please cite:

@software{palace2025,
  title = {PALACE: Platform for Automated LLMs Agentic Capabilities Evaluation},
  author = {Altieri, Massimiliano},
  year = {2025},
  institution = {European Commission Joint Research Centre},
  url = {https://github.com/palace-ai/palace-eval},
  license = {EUPL-1.2}
}

Support

• Issues: GitHub Issues
• Email: [e-mail]