MazeBench

MazeBench is a world-building, play, and AI-agent benchmarking platform. Create persistent multi-room 3D worlds, navigate them yourself in the browser or terminal, and run coding agents or Prime Intellect Verifiers against the same JavaScript engine.

Local agent runs support Codex CLI and Claude Code. The Verifiers v1 integration supports both normal multi-turn chat-model runs and coding-agent harnesses.

Install from PyPI and launch

pip install mazebench     # or: uv tool install mazebench
mazebench launch          # serves the site and opens your browser

The wheel bundles the whole Node site/runtime; mazebench launch materializes a writable workspace at ~/.mazebench/site (drafts, run outputs, and account state live there) and serves it. Node.js must be installed; ffmpeg + a Chromium-family browser are needed for replay videos, and the codex / claude CLIs for agent runs. Options: mazebench launch port=3000 host=127.0.0.1 open=false. From a repo checkout, mazebench launch serves the checkout instead (same as npm run dev).

Releases are published by .github/workflows/publish.yml (PyPI trusted publishing) whenever a GitHub Release is created; see docs/packaging.md.

The local site: Play, Build, and Agent modes

npm run dev (or mazebench launch) serves the full site at http://localhost:3000 with three modes on the home page (the frontend design matches mazebench.com):

• Play Mode (/play) — play the master world (the 256-room world agents are benchmarked on), any local draft world, or downloaded online worlds.
• Build Mode (/build) — make and save your own worlds locally, without publishing anywhere. Each draft world is a real game directory under games/draft-* (gitignored), so the level editor (/author/<world>/<level>), world-map editor, play page, flyover, and the agent runner all work on it. You can also edit the master world itself (its levels live in games/maze/), copy the master world into a draft, and import/export worlds as JSON in the same mazebench-build-world-v1 format the hosted site uses.
• Agent Mode (/agent) — launch Codex CLI, Claude Code (your subscriptions, via the same runner as mazebench model=...), or Prime Verifiers runs from the browser, against the master world, any local draft, or a downloaded community world. Runs stream live: per-move actions, gems, the ASCII board, the agent's reasoning, the runner log, and the replay video when the run finishes. Run artifacts land under outputs/maze-local/site/<run-id>/.

For Prime agent harnesses, choose Custom and select any built-in discovered from the exact pinned Verifiers package. Codex and Claude Code can also be sent through Prime from their dedicated cards. Native MCP harnesses connect directly; Codex receives the same MCP server through generated CLI config; command-only harnesses such as mini-swe-agent and Terminus 2 receive a capability-scoped mazebench-game CLI. The harness program runs in a disposable Prime sandbox while maze state, checkpoint, source, and scoring stay in the trusted evaluator. Only sanitized start, observe, and action operations cross that boundary.

The Agent page does not need an example.toml. Launch choices, including the harness id and any exact harness version, are saved in each run's run.json and restored by Continue. TOML is an optional Verifiers CLI configuration format, not MazeBench's run persistence format. A documented optional example lives at environments/mazebench/example.toml.

World editor source of truth

MazeBench owns the complete world-editor frontend and shared site design tokens. The canonical pieces are public/site.css, public/author-shell.js, public/author-theme.css, public/author.js, and the shared play/render runtime they load. The Maze Jam hosted app copies these assets during its build and supplies only hosted configuration, persistence, accounts, and publishing behavior. Do not add a separate editor DOM or skin in Maze Jam; editor changes made here flow there through its asset-sync build.

Sync drafts with mazebench.com

Build Mode can connect to your account on the hosted site (https://dev.mazebench.com, soon mazebench.com): the local server holds a session token in data/remote.json and talks to the site server-to-server — push local drafts up, pull site drafts down, and download published community worlds for agent runs. Connect either via the browser link flow (/link-local on the hosted site) or by pasting your mazebench_session cookie value manually. Drafts stay drafts — publishing remains a deliberate action on the site.

The mazebench command

There is a small mazebench CLI that wraps every workflow. It is a launcher: it shells out to the repo's Node scripts (game engine, local-agent runner, replay/video) and, for the Prime path, to the prime/uv CLIs. Node.js is a prerequisite for all runs.

Install it from a checkout (recommended — the CLI needs the repo's Node scripts at runtime):

# with uv (this repo already uses a uv-managed .venv)
uv pip install -e .

# or with plain pip
pip install -e .

# add the Prime Intellect Verifiers integration
uv pip install -e ".[prime]"

Local agent runs execute inside a container by default so the agent is isolated from your filesystem. Build the image once (needs Docker):

mazebench build          # or: npm run maze:build-image  /  docker build -t mazebench-agent .

Then, without Prime, drive the maze with your own local agent and get a replay video out the other end (credentials are read from OPENAI_API_KEY / ANTHROPIC_API_KEY in your environment):

mazebench model=codex moves=10
mazebench model=claude moves=10 level=HxI

See Run with a local coding agent for details, and docs/packaging.md for publishing to PyPI.

Run with a local coding agent (no Prime)

This path uses your Codex or Claude Code auth. There is no Prime intercept server and no reward scoring — the agent just plays the game by shelling out to scripts/codex-play.js (a stateful CLI over scripts/maze-bridge.js), and the runner then renders a replay video from the session.

Prefer not to remember the flags? Run the interactive setup and pick each option with the arrow keys:

mazebench wizard            # or: npm run maze:wizard

The wizard's "Custom…" model choice lists your actual Codex models (read from ~/.codex/models_cache.json), then — for Codex — lets you pick the reasoning effort (low/medium/high/xhigh) and toggle Fast mode. You can also set these directly:

mazebench model=codex model_name=gpt-5.5 reasoning=xhigh codex_fast=true moves=10
mazebench model=claude model_name=opus moves=10

model_name is forwarded to codex -m / claude --model. reasoning maps to codex -c model_reasoning_effort (Codex: low/medium/high/xhigh) or to claude --effort (Claude Code: low/medium/high/xhigh/max); codex_fast=true maps to Codex's priority service tier.

One-liners:

# via the mazebench CLI
mazebench model=codex moves=10
mazebench model=claude moves=10

# equivalently, via npm
npm run maze:codex-local -- moves=10
npm run maze:claude-local -- moves=10

Common options (all accept key=value or --key value):

In ASCII mode, a lowered player lift uses > on top; a raised lift uses L on top, and both use l on their sides. Orange walls use O on top and o on their sides; o is a face character, not a lowered-wall state. Orange buttons are top-only surface attachments rendered as 8, with no side character. Pressing a button moves the wall geometry down one elevation. In JSON mode, lifts use player_lift_lowered/player_lift_raised; orange walls remain orange_wall and their elevation coordinate decreases by one while pressed.

Use dry_run=on to preview exactly what will run (the container/agent command and the full play prompt) without spending any tokens or needing Docker.

Isolation: runs happen in a container by default

Every local agent run executes inside a container (container=true, the default). Only the run's output directory is writable. The current maze runtime (scripts, server, public, and games/maze) is mounted read-only so a new run or branch immediately uses the latest ASCII, JSON, and gameplay behavior; the rest of your filesystem remains unavailable. This is an OS-level guarantee and does not depend on the agent CLI behaving.

• One-time build: mazebench build (or npm run maze:build-image). The image bundles Node, the maze runtime, a headless Chromium + ffmpeg (for vision and video), and the codex / claude CLIs. See the Dockerfile.
• Credentials (forwarded into the container as env, or mounted):
  • Codex: model=codex auto-mounts your Codex subscription login (~/.codex/auth.json, read-only) when it exists — no setup needed. Otherwise set OPENAI_API_KEY, or pass codex_auth=<path>.
  • Claude: on macOS, model=claude auto-detects your Claude Code subscription login from the Keychain and mounts just that credential (a short-lived, read-only temp file, deleted after the run) — no setup needed. Otherwise set ANTHROPIC_API_KEY or CLAUDE_CODE_OAUTH_TOKEN, or pass claude_auth=<path> to a .credentials.json.
  • Only the single credential is exposed — never your whole ~/.codex or ~/.claude (which hold history, memories and logs).
• What's mounted: only outputs/maze-local (writable, for artifacts). Nothing else on your disk is visible to the container. The network stays on because the agent needs to reach the model API.
• Escape hatch: container=false runs on the host instead, protected only by the per-CLI tools sandbox below (weaker — see the Codex read caveat).
• docker_bin=podman and image=<tag> override the runtime and image tag.

Sandboxing (tools)

Inside the container (or on the host with container=false), tools is a second layer. With tools=false (default) the agent is meant to only drive the maze helper; how strictly that is enforced depends on the agent (below). tools=true removes the guardrails (write files, run any command, use the network) — inside the container that is still safe for your host.

• Codex — On the host (container=false, tools=false), Codex runs under codex exec --sandbox workspace-write: writes confined to the run folder, network disabled by default. Caveat: Codex's OS sandbox does not restrict reads, so a determined agent could still read files it knows the path to. Inside the container, Codex's sandbox (bubblewrap) can't create user namespaces under Docker, so Codex runs with --dangerously-bypass-approvals-and-sandbox — the container itself is the sandbox. So for Codex-in-container, tools=false is a prompt-level instruction ("only play the maze"), not an OS guarantee; host isolation comes from the container. tools=true always uses --dangerously-bypass-approvals-and-sandbox.
• Claude Code (tools=false): runs under --permission-mode dontAsk with an allowlist of only the maze helper command (Bash(node <helper> *)), so every other tool — Read, Write, WebFetch, other Bash — is auto-denied. Claude also blocks command chaining per-subcommand, so the allowlist can't be widened with ; other-cmd. This is a true "maze-only" lockdown at the agent level. tools=true uses --permission-mode bypassPermissions.

Notes:

• In vision mode with tools=false, Claude is additionally allowed to read the rendered frames (Read(<run>/frames/**)) so it can see the maze. Under Codex's sandbox the headless-browser render may be blocked; if a frame fails, the helper falls back to the ASCII board for that turn (frame_error).
• These are the best guardrails each CLI offers; neither replaces running untrusted agents in a real VM/container if that is your threat model.

Text, JSON, and vision observations

By default the agent plays from the ASCII board (mode=text). In mode=vision the helper renders a perspective PNG of the current room each turn (via the same scripts/maze-render-frame.js renderer the Verifiers vision taskset uses), drops the ASCII board, and prints a frame_image path in the JSON. The agent — Codex CLI and Claude Code are both multimodal — opens that PNG to decide its move. Text and vision status do not include explicit player coordinates or elevation. Frames are saved under <run>/frames/.

In mode=json, the model instead receives objects grouped by type with [x,y,elevation] coordinates. Directional names such as ice_slope_up and puncher_left are camera-relative. By default, an object is included whenever at least one of its characters survives the final ASCII render; use omniscient=true for every object in the room. hide_names=true assigns stable random letter names for the run, while player and gem remain literal.

mazebench model=codex moves=10 mode=vision
mazebench model=claude moves=10 mode=vision vision_width=768 vision_height=768
npm run maze:codex-local -- moves=10 mode=vision
mazebench model=codex moves=10 mode=json omniscient=true hide_names=true

Vision mode boots a headless browser to render each frame, so it is noticeably slower than text mode and needs a Chromium-family browser (the same dependency as the replay video). A frame-render failure stops that observation rather than falling back to an ASCII board.

Each run writes a timestamped directory under outputs/maze-local/<model>/:

• session.json — full state and per-action replay
• actions.jsonl — per-turn action log
• scorecard.json, maze_scorecard.json, maze_actions.txt
• maze_replay.mp4 — the replay video (unless video=off)
• reasoning.json, agent.log, agent-events.jsonl — the agent's reasoning (see below)

Scorecards are runner-only artifacts generated after Codex or Claude Code exits. They are not exposed as CLI or MCP game controls and never appear in a model observation. JSON is the only mode that includes explicit player coordinates.

Reasoning logs

For both model=codex and model=claude, the runner captures the agent's structured event stream (codex exec --json / claude -p --output-format stream-json) and distills a per-move reasoning log. reasoning.json is exactly what you'd want to skim:

[
  { "move": 1, "action": "right", "reasoning": "The exit is east, so...",
    "moved": true, "gems": 0, "room": "level_HxI" }
]

The reasoning for each move is the agent's commentary (Codex agent_message / Claude text blocks, plus Claude thinking blocks if extended thinking is on) since its previous action. agent.log is the same trace in human-readable form, agent-events.jsonl is the raw event stream, and the runner prints a per-move summary to the terminal at the end of the run.

Prerequisites: Node.js, plus ffmpeg and a Chromium-family browser for the video (already used by npm run maze:replay). The agent binary must be on your PATH — codex for model=codex, claude for model=claude — or pass codex_bin=/claude_bin= with an explicit path.

To (re)build a video from any finished run or Prime eval directory:

mazebench replay outputs/maze-local/codex/<run-dir>/
# or
npm run maze:replay -- outputs/maze-local/codex/<run-dir>/

1. Set up the web app with Node

Install dependencies, run tests, then start the local server:

npm install
npm test
npm run dev

Open the app at:

http://localhost:3000

2. Run the game in the terminal, see the scorecard, and save video

Start an interactive terminal game:

npm run maze:terminal

Controls:

• Arrow keys: move
• W / S: pitch camera up / down
• A / D: yaw camera left / right
• z or u: undo
• r: reset
• q: quit and print the scorecard

Interactive runs write replay files under outputs/maze-terminal/<timestamp>/. When the run ends, answer the video prompt to save maze_replay.mp4.

For a non-interactive run that saves the scorecard and video:

npm run maze:terminal -- --moves UDLR --once --record-replay --video --fast --draft --fps 20 --width 400 --height 400

The output folder contains:

• maze_scorecard.json
• maze_actions.txt
• maze_replay.json
• results.jsonl
• maze_replay.mp4 when video is enabled

3. Run with Prime Intellect Verifiers and see results

The mazebench prime subcommands wrap the steps below (they print the exact underlying command before running it):

mazebench prime install                                   # prime env install mazebench
mazebench prime eval   model=openai/gpt-5-nano n=1 r=1    # normal chat-model eval
mazebench prime codex  model=openai/gpt-5-codex           # Codex CLI via Verifiers (section 4)
mazebench prime vision model=openai/gpt-4.1-mini          # vision observations (section 5)

Install uv if you do not already have it:

curl -LsSf https://astral.sh/uv/install.sh | sh

On macOS, Homebrew also works:

brew install uv

Install the local environment from ./environments/mazebench:

prime env install mazebench

Run a small eval. MazeBench is a Verifiers v1 taskset, so run it with the v1 eval CLI via uv (from the environment directory) — not prime eval run, which is the legacy env-module loader and can't load a v1 taskset:

cd environments/mazebench
uv run eval mazebench -m openai/gpt-5-nano -n 1 -r 1 --max-turns 20 --rich false

--max-turns is the per-rollout move budget (how many moves the agent gets before the rollout stops), -n the number of examples, -r the rollouts per example. Results save under environments/mazebench/outputs/. Agent Mode on the website launches exactly this command for a Prime run.

The environment uses a full 40-character Verifiers Git revision. A scheduled compatibility workflow proposes exact-pin updates only after rediscovering every built-in harness and passing the adapter and trust-boundary certification.

Use your own configured model after -m, or omit -m to use your Prime default. The terminal prints the run summary. MazeBench v1 stores replay artifacts under info.maze_actions, info.maze_scorecard, and info.maze_replay in results.jsonl.

View saved evals:

prime eval view

Export scorecard files and a replay video from a saved eval directory:

npm run maze:replay -- environments/mazebench/outputs/evals/<model>/<run-id>

4. Run Codex through Verifiers v1

MazeBench keeps the mazebench_codex v1 taskset id as a compatibility alias. It now uses the external mazebench-tools task and the Prime-sandboxed Codex MCP adapter, so Codex never receives the checkout or evaluator paths.

cd environments/mazebench
uv run eval mazebench_codex \
  --harness.id mazebench_codex_harness \
  --harness.runtime.type prime \
  --harness.runtime.image node:24-bookworm-slim \
  -m openai/gpt-5 \
  -n 1 -r 1 \
  --taskset.max-actions 100 \
  --max-turns 40 \
  --rich false

--taskset.max-actions is the maze action budget. --max-turns is the framework-enforced model-turn budget. Prime provides both the disposable runtime and inference interception; MazeBench retains authoritative state and scoring.

Export the saved v1 trace to scorecard files and video the same way as normal MazeBench evals:

npm run maze:replay -- environments/mazebench/outputs/evals/<model>/<run-id>

5. Run vision observations

The normal mazebench taskset can send perspective PNG observations instead of ASCII boards:

cd environments/mazebench
uv run eval mazebench \
  -m openai/gpt-4.1-mini \
  -n 1 -r 1 \
  --taskset.observation-mode vision \
  --taskset.vision-width 512 \
  --taskset.vision-height 512 \
  --max-turns 8 \
  --rich false

Vision mode keeps the same JS game state, allowed commands, terminal conditions, and reward functions as ASCII mode. It sends a short text status plus a perspective image of the current room, with no ASCII board.

License

MIT. See LICENSE.