Can LLM agents coordinate in long-horizon, open-ended tasks?

🌐 Website · 🏆 Leaderboard · ⚡ Evaluate an LLM · Submit a result · 📄 Paper · 🤗 Models

Alem is a JAX benchmark for open-ended multi-agent coordination. Building on Craftax and Multi-Agent Craftax / Craftax-Coop, Alem introduces procedurally generated coordination tasks, soft specialisation, communication, and controllable coordination difficulty into a long-horizon survival world with exploration, crafting, trading, and combat. The same world is exposed through symbolic, pixel, and text interfaces, making it usable by MARL agents, language agents, and humans.

Alem means world in Amharic.

_{93 achievements · 27 coordination goals · 3 difficulty tiers · 9 dungeon levels · episodes up to 10,000 steps · pure JAX}

Contents

RL Playing · LLM Playing · Install · Quick Start · Evaluate an LLM · Configure · RL Agents · Baselines · Human Play · Docker · Package Layout · Development · RL vs LLM Interfaces · Reproduce the Paper · Submit to the Leaderboard · Errata · Contributing · Citation · License

RL Agents Playing

A team of MARL agents controlling the three players from symbolic observations, each acting from its own egocentric view.

Fast to train end-to-end in JAX. Full MARL training code and reference baselines live in baselines/; see Baselines.

LLM Agents Playing

The same world through the text interface. Each agent gets its own observation, broadcasts a free-form message to teammates every step and stores important information in scratchpad memory.

_{Gemini 3.1 Pro (medium). THINKING = the agent's private plan; MESSAGE = what it broadcasts to the team. The view follows whichever agent is speaking; reasoning is typed out for readability, not the agents' real decision speed.}

Three agents negotiate a synchronous action. They plan ahead, model each other, lock in a shared step, and then execute it together:

• Plans ahead. Each agent commits to a turn-indexed plan (Step 5 Move · Step 6–7 Noop · Step 8 DO) and waits in position.
• Theory of mind. A0 predicts a teammate's move will break the plan ("A2, your Step-6 DO will fail because A1 changed to Step 8"), and A1 catches the clash: "A0 and A2 sent conflicting times — EVERYONE DO ON STEP 8!"
• Coordinates out loud. All three line up and DO the 3-agent tree on the same step.

See Evaluate an LLM to run it yourself.

You are Agent {id} ({role}) in a {num_agents}-agent cooperative survival game. Your goal is to gather resources, craft gear, fight monsters, and descend through {num_levels} dungeon levels, while coordinating with teammates. You must survive — if your health reaches zero, you die, and if all agents die the game ends. Maximize achievements while alive.

[ … full game rules: movement & facing, Do/interaction, crafting recipes, roles & specialist penalties, coordination (sync / handover / construction), the resource chain, and the {num_achievements} achievements …]

<output_format>
1. (Required) Exactly one action from the available list:
   <action>YOUR_CHOSEN_ACTION</action>
2. (Optional) Broadcast to teammates, up to {comm_char_limit} chars:
   <communication>YOUR_MESSAGE</communication>
3. (Optional) Private notes, up to {scratchpad_char_limit} chars — not shared; your only memory:
   <scratchpad>YOUR_NOTES</scratchpad>
Token budget: {token_budget} tokens for the full response (including reasoning).
</output_format>

Step: {step}/{max_steps} ({steps_remaining} remaining, ends early if all agents die)
Position: (x={x}, y={y})
Role: {role}
Location: {dungeon_level}
Achievements: {unlocked}/{total} ({locked} unlock later)

You see:
- {object} {relative_position} (x={x}, y={y})
  ...one line per visible object...

Facing: {direction}. Do target: {object_in_front} (x={x}, y={y}).

Coordination:
- {object} (x={x}, y={y}): {how_this_object_must_be_coordinated}
  ...one line per coordination-relevant object...

Teammates:
Agent {id} ({role}): {relative_position} (x={x}, y={y}), health={hp}
  ...one line per teammate...

Your status: health {hp}, food {food}, drink {drink}, energy {energy}, mana {mana}, xp {xp}
Available actions: {legal_actions_this_step}

Step: 0/10000 (10000 remaining, ends early if all agents die)
Position: (x=24, y=24)
Role: warrior
Location: Overworld (surface)
Achievements: 0/93 (39 unlock later)

You see:
- stone 5 steps east (x=29, y=24)
- tree 1 step north and 2 steps west (x=22, y=23)
- construction_site 2 steps north (x=24, y=22)
- iron 3 steps south and 5 steps east (x=29, y=27)

Facing: north. Do target: grass (x=24, y=23).

Coordination:
- construction_site (x=24, y=22): requires 3 agents to Build simultaneously (fails alone).
- tree (x=26, y=23): one agent begins, another completes it within 6 steps (handover).
- stone (x=20, y=21): works solo, but a bonus when 3 agents Do simultaneously.

Teammates:
Agent 1 (forager): 1 step east (x=25, y=24), health=9
Agent 2 (miner): 1 step south (x=24, y=25), health=9

Your status: health 9, food 9, drink 9, energy 9, mana 9, xp 0
Available actions: Noop, Move {West,East,North,South}, Do, Sleep, Rest, Request {Food,Drink,Wood,Stone,Iron,Coal,Diamond,Ruby,Sapphire}

Install

uv venv --python 3.12
uv pip install alem-env # latest release from PyPI     

Or from source for development (editable install):

uv pip install -e .

Optional extras (work with either alem-env or -e .):

uv pip install -e ".[llm]"             # OpenAI-compatible LLM interface
uv pip install -e ".[play]"            # pygame human-play example
uv pip install -e ".[gpu]"             # NVIDIA CUDA 12 JAX wheels
uv pip install -e ".[baselines-rl]"    # JAX MARL trainers (see Baselines)
uv pip install -e ".[baselines-llm]"   # LLM-agent evaluation harness
uv pip install -e ".[dev]"             # pytest, ruff, jaxtyping

Plain pip also works (no uv required):

pip install -e .
pip install -e ".[gpu]"

Or pip install alem-env.

Quick Start

import jax
from alem.alem_env import make_alem_env_from_name

env = make_alem_env_from_name("Alem-Coop-Symbolic")
obs, state = env.reset(jax.random.PRNGKey(0))

rng_act = jax.random.split(jax.random.PRNGKey(1), env.num_agents)
actions = {agent: env.action_space(agent).sample(rng_act[i]) for i, agent in enumerate(env.agents)}

obs, state, rewards, dones, infos = env.step(jax.random.PRNGKey(2), state, actions)

Available environments:

Evaluate an LLM

Alem is an open leaderboard for multi-agent coordination: evaluate any OpenAI-compatible model and submit your score. Three agents play as a team through the text interface: each reasons privately, broadcasts a message to teammates, and keeps a scratchpad.

uv pip install -e ".[baselines-llm]"   # the LLM evaluation harness

Local model (vLLM or any OpenAI-compatible server). The same MODEL_ID drives all three agents:

# Install vLLM in a SEPARATE env; it pins its own torch/CUDA build that clashes with this repo's jax
uv venv --python 3.12 .venv-vllm
source ~/.venv-vllm/bin/activate
uv pip install vllm --torch-backend=auto

# Terminal 1: serve your model
vllm serve meta-llama/Llama-3.2-1B-Instruct --port 8000 

# Terminal 2: evaluate a 3-agent team on all leaderboard difficulties (from this repo's main env)
scripts/run_llm_eval.sh meta-llama/Llama-3.2-1B-Instruct \
    --base-url http://localhost:8000/v1 --episodes 20 --difficulty easy,medium,hard

Any OpenAI-compatible server works; vLLM is just the common choice for open models. See vLLM install docs for GPU/CPU build options.

Hosted API (OpenAI / Anthropic / Gemini / …):

export OPENAI_API_KEY=sk-...
python baselines/llm/eval_alem.py \
    clients.0.client_name=openai clients.1.client_name=openai clients.2.client_name=openai \
    clients.0.model_id=gpt-4o-mini clients.1.model_id=gpt-4o-mini clients.2.model_id=gpt-4o-mini \
    eval.num_episodes.alem=20

Swap client_name to anthropic, gemini, nvidia, or xai (and set the matching API key). Runs use the default robust_all agent + specific_collaborative prompt, the paper setup.

Try it before spending tokens. Run a cheap 5-step smoke test, or preview the text observations with no model at all:

scripts/smoke_llm.sh meta-llama/Llama-3.2-1B-Instruct --base-url http://localhost:8000/v1 --steps 5 --coord easy
uv run python examples/llm_text_smoke.py --coord easy --show-affordances   # no model calls

Prefer Docker? A single docker run serves your model with vLLM and runs the 3-agent eval, with no local install. See Docker → Evaluate an LLM.

→ Full harness reference: agent types, prompt modes, providers, per-agent configs, and metrics: baselines/llm/README.md. To publish a result: SUBMISSION.md.

Configure

from alem.alem_coop.alem_state import EnvParams, StaticEnvParams, get_coordination_params

env_params = EnvParams().replace(
    **get_coordination_params("easy"),
    soft_specialization=True,
    shared_reward=False,
)
static_env_params = StaticEnvParams(player_count=3, num_comm_channels=4)

env = make_alem_env_from_name(
    "Alem-Coop-Symbolic",
    env_params=env_params,
    static_env_params=static_env_params,
)

Coordination difficulty can be "none", "easy", "medium", "hard", or a numeric alpha in [0, 1].

RL Agents

A minimal framework-free rollout using symbolic observations and legal action masks:

uv run python examples/random_rl_agent.py --coord easy --steps 100
uv run python examples/random_rl_agent.py --players 2 --coord hard --steps 200

The example uses a jitted lax.scan loop and can serve as a template for custom policies. Full MARL training recipes (IPPO, HyperMARL-IPPO, MAPPO, PQN-VDN) live in baselines/; see Baselines.

Baselines

Reference MARL training code and the LLM-agent evaluation harness live under baselines/. Each RL algorithm is a single self-contained file (CleanRL-style) with a matching Hydra config; the LLM harness (derived from BALROG) drives 3 language agents through the text interface.

uv pip install -e ".[baselines-rl]"    # JAX MARL trainers (IPPO / MAPPO / PQN-VDN / HyperMARL)
uv pip install -e ".[baselines-llm]"   # LLM-agent evaluation harness

cd baselines && python ippo_rnn.py TOTAL_TIMESTEPS=10000           # train an RL policy
cd baselines/llm && python eval_alem.py clients.0.client_name=openai \
    clients.1.client_name=openai clients.2.client_name=openai      # evaluate an LLM team

Human Play

Install the optional play dependencies first:

uv pip install -e ".[play]"

uv run python examples/play_alem.py
uv run python examples/play_alem.py --players 3 --coord easy --seed 42
uv run python examples/play_alem.py --players 2 --coord hard --god

The game advances after all players have chosen an action.

Docker

Three images, each built from the repo root. Two of them run a full benchmark job in a single docker run; the container installs nothing and starts working immediately:

Both alem-llm and alem-rl need nvidia-container-toolkit on the host and --gpus at runtime.

Evaluate an LLM in one command

docker build -f docker/Dockerfile.llm -t alem-llm .

# Serves the model with vLLM, waits until ready, then evaluates a 3-agent team.
# Only MODEL_ID is required; mount the HF cache so models are downloaded once.
docker run --rm --gpus all --shm-size=16g \
    -e MODEL_ID=meta-llama/Llama-3.2-1B-Instruct \
    -e HF_TOKEN=$HF_TOKEN \
    -v ~/.cache/huggingface:/app/.cache/huggingface \
    alem-llm eval.num_episodes.alem=5

Trailing arguments are Hydra overrides for the eval (e.g. eval.num_episodes.alem=5, alem.coordination_difficulty=hard). For a large model, shard across GPUs with -e VLLM_EXTRA_ARGS="--tensor-parallel-size 4" and a bigger --shm-size. To log to W&B, pass -e WANDB_API_KEY=... -e WANDB_MODE=online. Full knobs are in the header of docker/Dockerfile.llm.

Train an RL policy in one command

docker build -f docker/Dockerfile.rl -t alem-rl .

docker run --rm --gpus all -v "$PWD/outputs:/app/outputs" \
    alem-rl ippo_rnn TOTAL_TIMESTEPS=1e4 NUM_ENVS=16 SEED=0

The first argument is the trainer (ippo_rnn, ippo_rnn_nops, ippo_hypermarl_rnn, mappo_rnn, pqn_vdn_rnn); the rest are Hydra overrides. Run with no arguments to list the available trainers. Checkpoints and GIFs land in the mounted outputs/.

Environment-only image

# CPU (default)
docker build -f docker/Dockerfile.env -t alem-env .

# GPU: NVIDIA CUDA 12
docker build -f docker/Dockerfile.env --build-arg ALEM_ACCELERATOR=cuda12 -t alem-env:gpu .

The image uses the system Python (UV_SYSTEM_PYTHON=1), so inside a container you call python directly, no uv run prefix needed.

docker run --rm alem-env                                           # smoke test (default CMD)
docker run --rm --gpus all alem-env:gpu                            # GPU smoke test
docker run --rm alem-env python examples/random_rl_agent.py --steps 20
docker run --rm alem-env python examples/llm_text_smoke.py --coord easy
docker run --rm -it alem-env bash                                  # interactive shell

Human play is easiest natively: uv pip install -e ".[play]" then uv run python examples/play_alem.py. Pygame opens a real window with no display plumbing.

# Build with the play extra (adds pygame + SDL/X11 runtime libs)
docker build -f docker/Dockerfile.env --build-arg ALEM_EXTRAS=play -t alem-env:play .

Package Layout

Development

uv pip install -e ".[dev]"   # pytest, ruff, jaxtyping
uv run pytest alem/tests/    # run the test suite

uv run ruff check .          # lint
uv run ruff format --check . # verify formatting

uv run ruff check --fix .    # apply safe lint fixes
uv run ruff format .         # format the code

RL vs LLM Interfaces

Both interfaces drive the same environment but are not directly comparable; treat cross-paradigm scores as indicative, not head-to-head.

(Request/Give resource transfers are ordinary actions in both.)

Text observations apply lightweight preprocessing, including compact local-state summaries (inspired by BALROG) and action affordances. See the language wrapper for details.

Reproduce the Paper

The paper's full experiments (the 13-LLM evaluation and the IPPO, HyperMARL-IPPO, MAPPO, PQN-VDN baselines) live in baselines/; see Baselines for launch commands and configs.

The paper's numbers were produced against Alem v0.1.0. For the exact settings to use when reporting an Alem number (seeds, episode count, metrics), see the canonical evaluation protocol.

Submit to the Leaderboard

Alem is an open benchmark. To add an LLM agent, MARL policy, or custom harness result to the leaderboard, follow SUBMISSION.md.

If your model is behind an OpenAI-compatible endpoint such as vLLM, the LLM path is one command:

scripts/run_llm_eval.sh YOUR_MODEL_ID --base-url http://localhost:8000/v1 --episodes 20 --difficulty easy,medium,hard

Use scripts/smoke_llm.sh YOUR_MODEL_ID --base-url http://localhost:8000/v1 --steps 5 --coord easy first for a cheap smoke test. Submissions should include enough detail to reproduce the run: model or algorithm, harness, coordination difficulty, seeds, metrics, evaluation date, and any non-default config. Pull requests are preferred, but issues and email submissions are also accepted.

Errata

Fixes that change environment behaviour are listed here, so numbers produced on different versions can be compared with the difference in mind. Report an Alem number together with the version it was produced on.

• Prior to v0.2.0 - ladder placement. The up-ladder and down-ladder positions were drawn independently and never checked against each other, so they could share a row and the up-ladders would replace some of the down-ladders. Affected levels could have fewer ways down than intended. This should be a minor issue for most runs, but it does change generated worlds, so results from before and after the fix are not directly comparable.

• Prior to v0.2.1 - lava bridging. Stone can now be placed into lava and mined into a walkable path, restoring Craftax's Fire Realm bridging mechanic without changing generated maps. This changes episode behaviour, but no agents have reached the Fire Realm yet, so it doesn't affect scores.

Contributing

Contributions are welcome: new baselines, bug fixes, docs, and coordination tasks. See CONTRIBUTING.md for the dev setup, lint/test workflow, and PR checklist, and CODE_OF_CONDUCT.md for community expectations.

Citation

@article{tessera2026alem,
  title   = {Benchmarking Open-Ended Multi-Agent Coordination in Language Agents},
  author  = {Tessera, {Kale-ab} Abebe and Szecsenyi, Andras and Barker, Cameron and
             Rutherford, Alexander and Paglieri, Davide and Scannell, Aidan and
             Gouk, Henry and Crowley, Elliot J. and Rockt\"{a}schel, Tim and
             Storkey, Amos},
  year    = {2026},
  url     = {https://arxiv.org/abs/2606.08340}
}

License

MIT. See LICENSE.