Skip to main content

boxd sandboxes for Inspect AI

boxd-inspect

A boxd sandbox for Inspect AI. Each sample runs on its own boxd machine, a full Ubuntu 24.04 VM, and the machine is deleted when the sample ends. Tasks written for Inspect's Docker sandbox run unchanged: their compose.yaml or Dockerfile runs with Docker inside the machine.

A boxd machine is a KVM microVM with its own kernel, sudo, Docker, Python, Node and Go. It boots in about a second, so you can run many samples in parallel without a Docker host of your own. Because each sample is a whole machine, the agent can build images, start containers and install system packages, and you can start every sample from a snapshot that already has your tools and images in it.

Install

pip install boxd-inspect

You'll need a boxd API key (sign up, then create a key in the console or with boxd auth keys create inspect):

export BOXD_API_KEY=bxd_...

When Inspect itself runs on a boxd machine you can skip the key: the machine's own credential works. Python 3.11 or later.

Installing the package registers the sandbox type boxd with Inspect. There is nothing to import in your task.

Quick start

# kernel.py
from inspect_ai import Task, task
from inspect_ai.dataset import Sample
from inspect_ai.scorer import includes
from inspect_ai.solver import generate, use_tools
from inspect_ai.tool import bash


@task
def kernel():
    return Task(
        dataset=[
            Sample(
                input="Run `uname -s` with the bash tool and tell me its output.",
                target="Linux",
            )
        ],
        solver=[use_tools(bash(timeout=60)), generate()],
        scorer=includes(),
        sandbox="boxd",
    )
pip install anthropic
export ANTHROPIC_API_KEY=...
inspect eval kernel.py --model anthropic/claude-haiku-5-5

Any model Inspect supports works.

Run an existing Docker task on boxd

Override the sandbox type, and the task's own compose.yaml or Dockerfile is used:

inspect eval my_task.py --sandbox boxd --model anthropic/claude-haiku-5-5

Or name the file in the task: sandbox=("boxd", "compose.yaml"). The compose file's directory (all of it except .git) goes to the machine, so build: contexts next to it work. Services named default or marked x-default: true are the default sandbox, as in Docker. Several services share one machine.

Without a compose file, commands run on the machine itself: Ubuntu 24.04, user boxd with passwordless sudo, Docker, Python, Node, Go and git preinstalled. The working directory is /home/boxd.

Configuration

Pass a BoxdSandboxEnvironmentConfig, or a JSON file with the same fields: sandbox=("boxd", "boxd.json").

from inspect_ai.util import SandboxEnvironmentSpec

from boxd_inspect import BoxdSandboxEnvironmentConfig

sandbox = SandboxEnvironmentSpec(
    "boxd",
    BoxdSandboxEnvironmentConfig(region="fra", vcpu=4, delete_after=3600),
)
Field Default Means
compose none A compose.yaml or Dockerfile to run with Docker inside the machine.
region BOXD_REGION, then your account's fra or nyc.
org BOXD_ORG, then the key's org The org the machines belong to.
vcpu the org's machine size 1, 2, 4 or 8. Memory follows: 4, 8, 16 or 32 GiB.
memory follows vcpu 4G, 8G, 16G or 32G.
from_snapshot none Start each sample from this snapshot, for example one with images already pulled.
isolated true The agent's commands get no peers on your org's private network and none of the in-machine boxd credentials, CLI or connected integrations your other machines have. Outbound internet still works. Turn it off only for code you trust.
working_dir /home/boxd, or the container's Where commands run and relative paths resolve.
name_prefix inspect Machine names are <prefix>-<task>-<sample>-<random>.
labels none Extra labels on every machine. Each machine also has inspect-sandbox=boxd, inspect-run and inspect-task.
delete_after none Seconds after its creation at which boxd deletes a machine, whatever it is doing. A safety net for evals that are killed. Keep it above your longest sample.
ready_timeout 180 Seconds to wait for a new machine.
compose_timeout 900 Seconds docker compose up may take, pulls and builds included.

The API key is read from BOXD_API_KEY (or BOXD_TOKEN) only. A key in the configuration would end up in the eval log.

Environment variable Means
BOXD_API_KEY The API key.
BOXD_REGION, BOXD_ORG The defaults for region and org. inspect sandbox cleanup boxd lists the machines of BOXD_ORG.
BOXD_INSPECT_CLEANUP_RUNNING Set to 1 and inspect sandbox cleanup boxd also deletes machines that are in use.

What is supported

Inspect On boxd
exec with input, cwd, env, timeout Yes.
exec(user=...), read_file(user=...), write_file(user=...) Yes. On the machine through sudo, in a service through docker compose exec --user.
read_file, write_file Yes. Missing directories are created. Permissions apply.
Sample files and setup Yes.
Several sandboxes per sample Yes, as services of one compose file in one machine.
connection() (for example the human agent) The boxd connect <machine> command, and the service as container.
inspect sandbox cleanup boxd [id] With an id: deletes that machine, if this package made it. Without: lists every machine with inspect-sandbox=boxd in the org, in all regions, and deletes the ones that are not in use. See Clean up.

Every check in Inspect's portable sandbox suite (inspect_ai.util._sandbox.self_check) passes on the machine and in Debian and Alpine compose services. A container running as root passes all but the three checks that Inspect's own Docker sandbox also expects root to fail.

timeout_retry and concurrency have no effect. Limit parallel machines with --max-sandboxes.

Errors

Configuration and key problems stop the eval before the first sample. Everything else fails the sample it happens in, and its machine is still deleted.

What happened What you get
A configuration value boxd doesn't accept (region, size, label, prefix, path) ValueError when the task starts, before any machine is made
The compose file's directory packs to more than 512 MiB ValueError when the task starts
A config that is not a BoxdSandboxEnvironmentConfig, ComposeConfig, compose file, Dockerfile or JSON file TypeError when the task starts
No API key, or boxd rejected it the boxd SDK's error (boxd.AuthenticationError for a rejected key) when the task starts
The machine didn't answer within ready_timeout TimeoutError; the sample fails and its machine is deleted
The compose project's default service didn't start RuntimeError with the end of the docker compose up output; the machine is deleted
A command ran past its timeout TimeoutError. The command and its child processes are stopped.
A command printed more than 10 MiB OutputLimitExceededError, and the command is stopped
A command that exists but can't be run PermissionError
read_file of a file over 100 MiB OutputLimitExceededError
read_file or write_file on a missing file, a directory, or without permission FileNotFoundError, IsADirectoryError or PermissionError
user= names a user that doesn't exist SandboxUserUnsupportedError
The machine was deleted while the sample ran SandboxUnavailableError
Deleting a machine failed a warning in the log; the machine is tried again when the task ends
inspect sandbox cleanup boxd <id> with an id that doesn't exist, or of a machine this package didn't make a message, exit code 1, nothing deleted
inspect sandbox cleanup boxd couldn't list a region, or a delete failed a message naming the region or machine, exit code 1; the rest is still cleaned up

A command that exits non-zero is not an error: exec returns its exit code, stdout and stderr, as in every Inspect sandbox.

Limits

Limit
Machines one per sample, shared by all services of its compose file
Machine size 1, 2, 4 or 8 vCPUs with 4 GiB of memory each, up to your organization's quota
Machine ready waits up to 180 seconds (ready_timeout); it usually takes about a second
docker compose up up to 900 seconds (compose_timeout), image pulls and builds included
Command output 10 MiB per stream, Inspect's limit for every sandbox
read_file 100 MiB, Inspect's limit for every sandbox
read_file, write_file 600 seconds each
Compose directory 512 MiB, sent to the machine as one archive
Machine name prefix 1 to 20 lowercase letters, digits or -, starting with a letter
Regions fra and nyc
inspect sandbox cleanup boxd one org (BOXD_ORG, or the key's), all its regions; machines that are running or starting are kept unless BOXD_INSPECT_CLEANUP_RUNNING=1

How it works

  • Installing the package adds an inspect_ai entry point. Inspect loads it at startup, and it registers BoxdSandboxEnvironment as the sandbox type boxd.
  • When the task starts, the configuration is checked and one API call checks the key, so a mistake shows up before any machine exists.
  • For each sample, sample_init creates a machine with the boxd Python SDK, waits until it takes commands, and makes a private temp directory on it. Auto-suspend is off for these machines, so a sample that thinks for a long time keeps its machine running.
  • Commands stream through the SDK's machines.exec. read_file and write_file run as small shell commands too, as the requested user, so the file's permissions apply. A write creates missing directories and writes through the existing file, so its owner and mode stay. Stdin and very long command lines go up as files first.
  • With a compose file, its directory is uploaded as one archive, and docker compose up --wait starts it inside the machine. Each running service becomes one sandbox, reached with docker compose exec.
  • When the sample ends, fails or is cancelled, its machine is deleted. Every machine id the eval made is recorded, and any left over is deleted by id when the eval ends. The record belongs to one eval: two evals in the same process never delete each other's machines.

Examples

Each one runs against real boxd. They use Claude (pip install anthropic, ANTHROPIC_API_KEY); pass any other model with --model.

  • examples/quickstart.py: the task above, one sample on one fresh machine.
  • examples/docker_task/: a task written for Inspect's Docker sandbox with its own Dockerfile, run unchanged with --sandbox boxd.
  • examples/docker_in_sample.py: three samples in parallel, each on its own machine, where the agent builds a Docker image, runs it with nginx and fetches the page.
  • examples/from_snapshot.py: prepare a machine once (install a tool, pull an image), save it as a snapshot, and start every sample from it.
inspect eval examples/quickstart.py --model anthropic/claude-haiku-5-5
inspect eval examples/docker_task/task.py --sandbox boxd --model anthropic/claude-haiku-5-5
inspect eval examples/docker_in_sample.py --model anthropic/claude-haiku-5-5
python examples/from_snapshot.py

Clean up

A sample's machine is deleted when the sample ends, also when it fails or is cancelled. Machines whose samples did not clean up are deleted, by id, when the task ends. With --no-sandbox-cleanup the machines stay and their ids are printed. If the eval process is killed, its machines stay. Set delete_after so boxd deletes them for you, or delete them yourself:

inspect sandbox cleanup boxd <id>                               # one machine
inspect sandbox cleanup boxd                                    # every machine that is not in use
BOXD_INSPECT_CLEANUP_RUNNING=1 inspect sandbox cleanup boxd     # the ones in use too

Without an id, the command lists every machine with the label inspect-sandbox=boxd in the org, in all regions. That includes machines of evals other people in the org are running right now. So it only deletes machines that are stopped, hibernated, in standby or failed, and keeps the ones that are running or starting. A machine of a killed eval is still running: delete it by id, or set BOXD_INSPECT_CLEANUP_RUNNING=1 when you know no eval is running in the org. In a terminal the command asks before it deletes. In a script or in CI it doesn't ask.

New machines start fresh, so a compose service's images are pulled for each sample. For big images, prepare a machine once, save it as a snapshot and pass from_snapshot (see examples/from_snapshot.py).

Development

Needs uv.

uv sync
uv run pytest                                      # offline, the boxd client is faked
uv run ruff check . && uv run ruff format --check .
uv run mypy src tests examples
BOXD_API_KEY=bxd_... uv run pytest -m live         # real machines

The live tests run Inspect's portable sandbox checks on a machine and in three compose setups, plus lifecycle and error tests. They create machines named t-inspect-*, labelled boxd-inspect=live-test, each with a one-hour delete_after, and delete them again. BOXD_INSPECT_TEST_REGION (default nyc) picks the region. Set BOXD_INSPECT_TEST_OTHER_REGION to a second region to also test cleanup across regions. They are skipped when BOXD_API_KEY is not set. No live test runs the cleanup without an id.

Resources

License

MIT

Metadata

Release files for boxd-inspect 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for boxd-inspect 0.1.0
File Size Uploaded Provenance Actions
boxd_inspect-0.1.0.tar.gz 37.4 kB Without provenance Details for boxd_inspect-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for boxd-inspect 0.1.0
File Interpreter ABI Platform Actions
boxd_inspect-0.1.0-py3-none-any.whl Python 3 none any Details for boxd_inspect-0.1.0-py3-none-any.whl

Total release size: 61.5 kB

Release files / boxd_inspect-0.1.0.tar.gz

Download URL boxd_inspect-0.1.0.tar.gz
Size 37.4 kB
Tags Source
SHA-256 checksum
How to use checksums
19dbe2748c1965261762439ba0ceffa7e79a0f1a9b0d2f26d5cee79785a398ae
BLAKE2b-256 checksum
How to use checksums
ff4867ef032fbc76550cbe3a891af86320f90dc9f2d9ec7c0827fba24ba7dbca
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.9

Release files / boxd_inspect-0.1.0-py3-none-any.whl

Download URL boxd_inspect-0.1.0-py3-none-any.whl
Size 24.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
9c0ec33a2d4b47d0fd5536ebddda268a6eddd4eba3099dd374339ffbd5c5913e
BLAKE2b-256 checksum
How to use checksums
3a49f25bf2345bdaab7e2067db32d5ad1bc31dcbf988de2416f07f9136498850
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.9

Security for boxd-inspect 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page