craftaxlm 0.1.0

No project description provided

Craftax LM

A wrapper around the Craftax agent benchmark, for evaluating digital agents.

Leaderboard

Craftax-Classic

LM	Algorithm	Reward (% max)	Code
gpt-4o-mini	ReAct	14.2	CraftaxLM_Baselines
33

Craftax-Full

LM	Algorithm	Reward (% max)	Code
gpt-4o-mini	ReAct	01.2	CraftaxLM_Baselines

Dev Instructions

pyenv virtualenv craftax_env
poetry install

When in doubt

from jax import debug
...
debug.breakpoint()

📚 Citation

To learn more about Craftax, check out the paper website here. To cite the underlying Craftax environment, see:

@inproceedings{matthews2024craftax,
    author={Michael Matthews and Michael Beukman and Benjamin Ellis and Mikayel Samvelyan and Matthew Jackson and Samuel Coward and Jakob Foerster},
    title = {Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning},
    booktitle = {International Conference on Machine Learning ({ICML})},
    year = {2024}
}

To cite the Crafter benchmark, see:

@article{hafner2021crafter,
  title={Benchmarking the Spectrum of Agent Capabilities},
  author={Danijar Hafner},
  year={2021},
  journal={arXiv preprint arXiv:2109.06780},
}