Skip to main content

PALACE

License: EUPL-1.2 Python 3.13+ PyPI version CI

PALACE logo

PALACE is an open benchmark format for LLM evaluation, with native support for agentic tasks.

Benchmarks should be portable. A PALACE benchmark is a self-contained folder with two JSON files: info.json (metadata and evaluation config) and tasks.json (the actual tasks). No framework lock-in, no Python decorators, no code to run. Just data that any tool can read.

palace-eval is the reference implementation.

Quick start

1. Install:

uv tool install palace-eval   # recommended
# or: pip install palace-eval

2. Configure your API:

palace config set url https://api.openai.com/v1
palace config set key sk-your-api-key
palace config set judge_model gpt-4o

3. Run your first evaluation:

palace download SimpleQA
palace run SimpleQA -m gpt-4o -l 10

That's it. Results go to ~/.cache/palace/results/.

Run palace config anytime to see your current settings.

Why another benchmark format?

Most benchmarks today are tied to specific frameworks. LM-eval-harness tasks are Python code. Inspect-AI uses @task decorators. If you want to run a benchmark, you need their infrastructure.

PALACE takes a different approach:

  • Self-contained: everything travels together (tasks, expected outputs, judge criteria, evaluation logic)
  • HuggingFace-distributed: download with palace download, standard hosting and discovery
  • Agentic-native: tool-using agents are a first-class task type
  • Framework-agnostic: palace-eval runs them, but so could anything else

We ship 27+ benchmarks covering reasoning, knowledge, safety, multilingual, multimodal, and agentic capabilities.

The format

A PALACE benchmark is just two JSON files:

SimpleQA/
├── info.json      # what kind of benchmark, how to evaluate
└── tasks.json     # the actual tasks

info.json defines the benchmark:

{
  "name": "SimpleQA",
  "id": "openai/SimpleQA", 
  "task_type": "QA",
  "category": "Knowledge"
}

tasks.json contains the tasks:

[
  {"id": "q1", "objective": "What is the capital of France?", "expected": "Paris"},
  {"id": "q2", "objective": "Who wrote Romeo and Juliet?", "expected": "Shakespeare"}
]

The format supports five task types (QA, Classification, Criteria Evaluation, Instruction Following, and Agentic), each with its own verification logic. See the full spec.

Configuration

Palace stores settings in ~/.config/palace/config.yaml. The quick start covers the essentials.

For CI/Docker, use environment variables instead. Run palace config env to see the mapping.

Included benchmarks

Category Benchmarks
Knowledge & Reasoning SimpleQA, HotpotQA, Humanity's Last Exam, GPQA Diamond, MUSR
Academic & Math MMLU, MMLU-Pro, MATH-500, AIME 2025, BBH, HellaSwag
Multilingual MMMLU (14 languages), MGSM, Belebele (122 languages)
Long Context BABILong (4k-128k), LongBench v2
Multimodal MMMU, MMMU Pro, VLSBench
Instruction Following IFEval
Agentic GAIA, AssistantBench
palace list                    # see all available
palace download MMLU           # download one
palace download --all          # download everything

Running evaluations

palace run MMLU -m gpt-4o
palace run MMLU -m gpt-4o -l 50           # limit to 50 tasks
palace run "GPQA Diamond" -m claude-sonnet
palace run SWE-bench -m o3-mini --agentic # agentic benchmark

Python API (for integration):

from palace import evaluate

evaluate(
    run_name="my-eval",
    url="https://api.openai.com/v1",
    token="sk-...",
    name="gpt-4o",
    tasklist="SimpleQA",
    limit=100,
)

Works with any OpenAI-compatible endpoint: OpenAI, Anthropic, Azure, Mistral, local deployments, etc.

Agentic evaluation

For benchmarks where the model uses tools (GAIA, AssistantBench, etc.), you need Vivarium, a sandboxed Docker runtime for agents.

pip install vivarium-ai

Vivarium starts automatically when you run an agentic tasklist. Requires Docker 24+.

Note: Vivarium is currently pending open-source release (expected within weeks). Contact massimiliano.altieri@ec.europa.eu for early access.

Creating your own benchmarks

palace init my-benchmark      # interactive wizard
palace validate my-benchmark  # check for errors
palace publish my-benchmark   # publish to HuggingFace

See Your First Benchmark for a walkthrough.

Documentation

Full docs at palace.pages.code.europa.eu/palace-eval.

Contributing

Issues and PRs welcome. See CONTRIBUTING.md.

License

Copyright 2025 European Union. Licensed under EUPL-1.2.

Developed by the European Commission Joint Research Centre.

Citation

@software{palace2025,
  title = {PALACE: An Open Benchmark Format for LLM Evaluation},
  author = {Altieri, Massimiliano},
  year = {2025},
  institution = {European Commission Joint Research Centre},
  url = {https://code.europa.eu/palace/palace-eval},
  license = {EUPL-1.2}
}

Contact

Metadata

Release files for palace-eval 1.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for palace-eval 1.2.0
File Size Uploaded Provenance Actions
palace_eval-1.2.0.tar.gz 144.4 kB Without provenance Details for palace_eval-1.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for palace-eval 1.2.0
File Interpreter ABI Platform Actions
palace_eval-1.2.0-py3-none-any.whl Python 3 none any Details for palace_eval-1.2.0-py3-none-any.whl

Total release size: 341.0 kB

Release files / palace_eval-1.2.0.tar.gz

Download URL palace_eval-1.2.0.tar.gz
Size 144.4 kB
Tags Source
SHA-256 checksum
How to use checksums
412fc60cacb9e6f6a9b214515bcb384a0c3be655f35745211652537cc028ac6a
BLAKE2b-256 checksum
How to use checksums
b156c019947345e685e5823164aa43c8ff599d40b705f38cf74126590e90a723
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.15

Release files / palace_eval-1.2.0-py3-none-any.whl

Download URL palace_eval-1.2.0-py3-none-any.whl
Size 196.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
98a85bb3be87eae6a0f9ba7547c18574b29df949fab436384cc340609de723c6
BLAKE2b-256 checksum
How to use checksums
4f1e4b3c6ee19963ca01132f175545fc28d4b0df067416832131db99cf0232a0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.15

Security for palace-eval 1.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page