cube-standard 0.1.0rc5

Common Unified Benchmark Environments

CUBE Standard

[!NOTE] CUBE is in active development (alpha). Interfaces may change. We welcome early adopters and contributors who want to shape the standard, not just use it. See our Roadmap and Contributing Guide.

Have a benchmark to contribute? Fill out this short form — no commitment required. Want to go deeper? Apply to join the core team.

This repo contains the code and documentation for the AI Alliance: CUBE Standard project, which standardizes benchmark wrapping so the community can wrap otherwise-incompatible benchmarks uniformly and use them everywhere.

CUBE Standard defines the protocol — the Tool, Task, Benchmark, Observation, and Action interfaces that any benchmark must implement. cube-harness is the evaluation runtime that runs agents against CUBE-compatible benchmarks.

Paper: arXiv:2603.15798

Principal developer: ServiceNow AI Research.

Components

CUBE Standard is organized into three layers:

Layer	Package	Description
Core	cube-standard (this repo)	interfaces: Tool, Task, Benchmark, Observation, Action
Resources	cube-resources/	Optional shared infrastructure (browser sessions, VM backends)
Tools	cube-tools/	Optional action executors (browser tools, computer tools)

Resources are pieces of shared infrastructure — e.g. a running browser instance or a VM — that are launched once and shared across tasks. Tools execute agent actions against that infrastructure.

Benchmark ──► TaskConfig ──► Task ──► Tool ──► Resource ──► Environment
                                ▲               (cube-tools)  (cube-resources)
                         cube-standard

See cube-resources/README.md and cube-tools/README.md for available implementations and usage examples.

Installation

Requires Python 3.12+. Install with uv:

uv add cube-standard

Or with pip:

pip install cube-standard

To include optional container backends:

# Docker support
uv add "cube-standard[docker]"

# Modal support
uv add "cube-standard[modal]"

# Daytona support
uv add "cube-standard[daytona]"

For development (includes test and lint tools):

git clone https://github.com/The-AI-Alliance/cube-standard
cd cube-standard
uv sync --extra dev

CLI commands

Command	What it does
cube init [NAME]	Scaffolds a new benchmark package from the built-in template
cube list	Lists all installed benchmarks registered under cube.benchmarks entry points
cube test NAME	Runs the debug suite and asserts reward == 1.0 on every debug task

For benchmark contributors

Fast path — copy the reference implementation, rename, and iterate:

cp -r examples/counter-cube my-bench
cd my-bench && uv sync
# Edit @tool_action decorated methods in src/*/tool.py
# Edit reset() and evaluate() in src/*/task.py
# Edit benchmark_metadata, task_metadata, task_config_class, _setup() and close() in src/*/benchmark.py
# expose get_debug_benchmark and get_debug_agent in src/*/debug.py
cube test my-bench

[!NOTE] cube test discovers benchmarks via the cube.benchmarks entry point group. The benchmark package must be installed (e.g. uv sync or pip install -e .) before running cube test, otherwise the benchmark will not be found.

Or scaffold from the template:

cube init my-bench    # scaffold a new benchmark package from the template
cd my-bench
uv sync
cube test my-bench    # run the debug compliance suite

See CONTRIBUTING.md for the five-layer architecture and implementation order.

Getting Involved

All contributions are welcome — open an issue, submit a PR, or wrap a new benchmark. See CONTRIBUTING.md for the development guide and RFC process.

Want to contribute a benchmark? Whether you're an original author or just a frequent user, fill out this short form to let us know. No commitment required — we'll follow up based on your interest and the benchmark's fit.

Want deeper involvement? Join the core team, shape the roadmap, and get credit for what you build. Apply here.

For general AI Alliance contribution guidelines, see the community repo and Code of Conduct.

All code contributions are licensed under the Apache 2.0 LICENSE (which is also in this repo, LICENSE.Apache-2.0).

All documentation contributions are licensed under the Creative Commons Attribution 4.0 International (which is also in this repo, LICENSE.CC-BY-4.0).

All data contributions are licensed under the Community Data License Agreement - Permissive - Version 2.0 (which is also in this repo, LICENSE.CDLA-2.0).

We use the "Developer Certificate of Origin" (DCO).

[!WARNING] Before you make any git commits with changes, understand what's required for DCO.

See the Alliance contributing guide section on DCO for details. In practical terms, supporting this requirement means you must use the -s flag with your git commit commands.

Pre-commit hooks (recommended)

This repo uses the pre-commit framework to run fast checks locally before you commit, including enforcing the DCO Signed-off-by line.

Install the hooks (you only need to do this once per clone):

pre-commit install --hook-type pre-commit --hook-type commit-msg

Run the checks on all files (optional, useful the first time):

pre-commit run --all-files

When committing, include your sign-off:

git commit -s -m "your message"