PALACE
PALACE is an open benchmark format for LLM evaluation, with native support for agentic tasks.
Benchmarks should be portable. A PALACE benchmark is a self-contained folder with two JSON files: info.json (metadata and evaluation config) and tasks.json (the actual tasks). No framework lock-in, no Python decorators, no code to run. Just data that any tool can read.
palace-eval is the reference implementation.
Quick start
1. Install:
uv tool install palace-eval # recommended
# or: pip install palace-eval
2. Configure your API:
palace config set url https://api.openai.com/v1
palace config set key sk-your-api-key
palace config set judge_model gpt-4o
3. Run your first evaluation:
palace download SimpleQA
palace run SimpleQA -m gpt-4o -l 10
That's it. Results go to ~/.cache/palace/results/.
Run palace config anytime to see your current settings.
Why another benchmark format?
Most benchmarks today are tied to specific frameworks. LM-eval-harness tasks are Python code. Inspect-AI uses @task decorators. If you want to run a benchmark, you need their infrastructure.
PALACE takes a different approach:
- Self-contained: everything travels together (tasks, expected outputs, judge criteria, evaluation logic)
- HuggingFace-distributed: download with
palace download, standard hosting and discovery - Agentic-native: tool-using agents are a first-class task type
- Framework-agnostic: palace-eval runs them, but so could anything else
We ship 27+ benchmarks covering reasoning, knowledge, safety, multilingual, multimodal, and agentic capabilities.
The format
A PALACE benchmark is just two JSON files:
SimpleQA/
├── info.json # what kind of benchmark, how to evaluate
└── tasks.json # the actual tasks
info.json defines the benchmark:
{
"name": "SimpleQA",
"id": "openai/SimpleQA",
"task_type": "QA",
"category": "Knowledge"
}
tasks.json contains the tasks:
[
{"id": "q1", "objective": "What is the capital of France?", "expected": "Paris"},
{"id": "q2", "objective": "Who wrote Romeo and Juliet?", "expected": "Shakespeare"}
]
The format supports five task types (QA, Classification, Criteria Evaluation, Instruction Following, and Agentic), each with its own verification logic. See the full spec.
Configuration
Palace stores settings in ~/.config/palace/config.yaml. The quick start covers the essentials.
For CI/Docker, use environment variables instead. Run palace config env to see the mapping.
Included benchmarks
| Category | Benchmarks |
|---|---|
| Knowledge & Reasoning | SimpleQA, HotpotQA, Humanity's Last Exam, GPQA Diamond, MUSR |
| Academic & Math | MMLU, MMLU-Pro, MATH-500, AIME 2025, BBH, HellaSwag |
| Multilingual | MMMLU (14 languages), MGSM, Belebele (122 languages) |
| Long Context | BABILong (4k-128k), LongBench v2 |
| Multimodal | MMMU, MMMU Pro, VLSBench |
| Instruction Following | IFEval |
| Agentic | GAIA, AssistantBench |
palace list # see all available
palace download MMLU # download one
palace download --all # download everything
Running evaluations
palace run MMLU -m gpt-4o
palace run MMLU -m gpt-4o -l 50 # limit to 50 tasks
palace run "GPQA Diamond" -m claude-sonnet
palace run SWE-bench -m o3-mini --agentic # agentic benchmark
Python API (for integration):
from palace import evaluate
evaluate(
run_name="my-eval",
url="https://api.openai.com/v1",
token="sk-...",
name="gpt-4o",
tasklist="SimpleQA",
limit=100,
)
Works with any OpenAI-compatible endpoint: OpenAI, Anthropic, Azure, Mistral, local deployments, etc.
Agentic evaluation
For benchmarks where the model uses tools (GAIA, AssistantBench, etc.), you need Vivarium, a sandboxed Docker runtime for agents.
pip install vivarium-ai
Vivarium starts automatically when you run an agentic tasklist. Requires Docker 24+.
Note: Vivarium is currently pending open-source release (expected within weeks). Contact massimiliano.altieri@ec.europa.eu for early access.
Creating your own benchmarks
palace init my-benchmark # interactive wizard
palace validate my-benchmark # check for errors
palace publish my-benchmark # publish to HuggingFace
See Your First Benchmark for a walkthrough.
Documentation
Full docs at palace.pages.code.europa.eu/palace-eval.
Contributing
Issues and PRs welcome. See CONTRIBUTING.md.
License
Copyright 2025 European Union. Licensed under EUPL-1.2.
Developed by the European Commission Joint Research Centre.
Citation
@software{palace2025,
title = {PALACE: An Open Benchmark Format for LLM Evaluation},
author = {Altieri, Massimiliano},
year = {2025},
institution = {European Commission Joint Research Centre},
url = {https://code.europa.eu/palace/palace-eval},
license = {EUPL-1.2}
}
Contact
- Issues: GitHub or GitLab
- Email: massimiliano.altieri@ec.europa.eu
Metadata
Release files for palace-eval 1.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| palace_eval-1.2.0.tar.gz | 144.4 kB | Without provenance | Details for palace_eval-1.2.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| palace_eval-1.2.0-py3-none-any.whl | Python 3 | none | any | Details for palace_eval-1.2.0-py3-none-any.whl |
Total release size: 341.0 kB
Release files / palace_eval-1.2.0.tar.gz
| Download URL | palace_eval-1.2.0.tar.gz |
|---|---|
| Size | 144.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
412fc60cacb9e6f6a9b214515bcb384a0c3be655f35745211652537cc028ac6a
|
|
BLAKE2b-256 checksum How to use checksums |
b156c019947345e685e5823164aa43c8ff599d40b705f38cf74126590e90a723
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.15
|
Release files / palace_eval-1.2.0-py3-none-any.whl
| Download URL | palace_eval-1.2.0-py3-none-any.whl |
|---|---|
| Size | 196.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
98a85bb3be87eae6a0f9ba7547c18574b29df949fab436384cc340609de723c6
|
|
BLAKE2b-256 checksum How to use checksums |
4f1e4b3c6ee19963ca01132f175545fc28d4b0df067416832131db99cf0232a0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.15
|
Security for palace-eval 1.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.