Craftax LM
A wrapper around the Craftax agent benchmark, for evaluating digital agents over extremely long time horizons.
Craftax-Classic
| LM | Algorithm | Score (% max) | Code |
|---|---|---|---|
| claude-3-7-sonnet-latest (default) | ReAct | 18.0 | |
| claude-3-5-sonnet-20241022 | ReAct | 17.8 | |
| claude-3-5-sonnet-20240620 | ReAct | 15.7 | |
| o3-mini | ReAct | 12.6 | |
| gpt-4o | ReAct | 7.0 |
- Note - this is a limited evaluation where trajectories are terminated after 30 api calls, or roughly 150 in-game steps. 10 trajectories are rolled-out, yielding a log-weighted score as per the Crafter paper. Reproducible code forthcoming.
Usage
First, download the package with pip install craftaxlm. Next, import the agent-computer interface of your choice via
from craftaxlm import CraftaxACI, CraftaxClassicACI
This package is early in development, so for implementation examples, please refer to the baseline ReAct implementation
Leaderboard
In order to make experiments reasonable to run across a range of LMs, currently the leaderboard evaluates agents in the following manner:
- Five rollouts are sampled from the agent, with a hard cap of 300 actions per rollout.
- The agent is evaluated using a modified version of the original Crafter score -
where P(1_achievement_obtained) is the probability of the achievement being obtained in a single rollout. The key idea is that incremental progress towards difficult achievements ought to weigh more heavily in the score.sum(ln(1 + P(1_achievement_obtained)) for achievement in achievements) / (sum(ln(2) * len(achievements)))
Craftax-Full
| LM | Algorithm | Score (% max) | Code |
|---|
Dev Instructions
pyenv virtualenv craftax_env
poetry install
When in doubt
from jax import debug
...
debug.breakpoint()
📚 Citation
To learn more about Craftax, check out the paper website here. To cite the underlying Craftax environment, see:
@inproceedings{matthews2024craftax,
author={Michael Matthews and Michael Beukman and Benjamin Ellis and Mikayel Samvelyan and Matthew Jackson and Samuel Coward and Jakob Foerster},
title = {Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning},
booktitle = {International Conference on Machine Learning ({ICML})},
year = {2024}
}
To cite the Crafter benchmark, see:
@article{hafner2021crafter,
title={Benchmarking the Spectrum of Agent Capabilities},
author={Danijar Hafner},
year={2021},
journal={arXiv preprint arXiv:2109.06780},
}
Contributing
Setup
uv venv craftaxlm-dev
source craftaxlm-dev/bin/activate
uv sync
uv run ruff format .
Help Wanted
- General code quality suggestions or improvements. Especially those that improve speed or reduce tokens.
- PRs to fix issues or add afforances that help your LM agent perform well
- Leaderboard submissions that demonstrate improved performance using algorithms for learning from data
1 maintainer
Metadata
1 maintainer
Release files for craftaxlm 0.0.37
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| craftaxlm-0.0.37.tar.gz | 32.1 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| craftaxlm-0.0.37-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 64.2 kB
Release files / craftaxlm-0.0.37.tar.gz
| Download URL | craftaxlm-0.0.37.tar.gz |
|---|---|
| Size | 32.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
8a2ab5373025e33f2750032cc91e8e64082b82dea257bb3d75a7d65ca0346d0e
|
|
BLAKE2b-256 checksum How to use checksums |
0db2b35f3705c340d43baca25094ae8402b8072704212feedf37edd52d4341e8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.11.8
|
Release files / craftaxlm-0.0.37-py3-none-any.whl
| Download URL | craftaxlm-0.0.37-py3-none-any.whl |
|---|---|
| Size | 32.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
65e1b458308203dd5d0571a8ca20c48315a3eecb74026b2da47ce24cb8009305
|
|
BLAKE2b-256 checksum How to use checksums |
667287906428ce8d33c26418175e0f472bb1e1adb8c66f6df344d6aa3e83d4bd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.11.8
|
Security for craftaxlm 0.0.37
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
