Skip to main content

agent-sandbox-harbor

A Harbor environment plugin that runs Harbor benchmarks (Terminal-Bench, SWE-bench, custom datasets) on Agent Sandbox pre-warmed pools — no fork of Harbor required.

Highlights:

  • Zero Harbor source changes. Plugs into Harbor via the official --environment-import-path extension point.
  • Skips Template Build. Agent Sandbox uses a pre-warmed Pod pool with in-place image swap, so the per-task Template Build step that E2B / Novita require is replaced by a single POST /v1/sandboxes call.
  • Internal-mirror friendly. A configurable image-prefix rewrites docker.io/... to your private Distribution / Harbor registry.
  • Bring-your-own image. An optional task-name → image map (AGBX_IMAGE_MAP) lets you run pre-built images for any dataset — including ones whose task.toml has no docker_image (e.g. SWE-bench, where the task is a Dockerfile).

Installation

pip install 'harbor[e2b]' agent-sandbox-harbor

The plugin pulls agent-sandbox-e2b as a hard dependency (it calls patch_e2b() at import). harbor is an optional peer dependency, so the package can be inspected / unit-tested without it; in real usage you install harbor[e2b] yourself.

Quick start

# 1. Set credentials (one-off)
cat > agentbox.env <<'EOF'
E2B_API_KEY=agbx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
E2B_DOMAIN=agent-sandbox-data-plane.example.com/agent-sandbox/api/data
E2B_API_URL=https://agent-sandbox-data-plane.example.com/agent-sandbox/api/e2b
AGBX_CLUSTER_ID=cluster-a
AGBX_POOL_NAME=terminal-bench-pool
AGBX_IMAGE_PREFIX=registry.internal/agent-sandbox
EOF

# 2. Run Harbor (use the plugin via the official --environment-import-path flag)
harbor run \
  -d terminal-bench@2.0 \
  -a oracle \
  --environment-import-path agent_sandbox_harbor:AgentSandboxEnvironment \
  -n 16 -y \
  --env-file agentbox.env

Configuration

Variable Required Description
E2B_API_KEY yes Agent Sandbox API key (agbx_...).
AGBX_ENV_NAME unless profiles specify it Existing SandboxEnv to run tasks in.
AGBX_POOL_NAME no Legacy alias for AGBX_ENV_NAME; both must agree if set.
AGBX_POOL_PROFILES no JSON capacity profiles selecting an existing Env and scaling group; see below.
AGBX_SCALING_GROUP no Exact existing group name. With profiles, restrict selection to this group and validate capacity; without profiles, pin placement only.
AGBX_DIND_ENV_NAME for Compose, unless profiles specify it Existing Docker-in-Docker SandboxEnv. AGBX_DIND_POOL_NAME is its legacy alias.
AGBX_DIND_POOL_PROFILES no Separate capacity profiles for Compose's outer DinD sandbox.
AGBX_DIND_SCALING_GROUP no Fixed group restriction for the DinD environment, independent of ordinary-task routing.
E2B_DOMAIN no Data-plane gateway, host[:port][/path]. Default is the in-cluster service.
E2B_API_URL no E2B-compatible control-plane URL, including scheme.
AGBX_CLUSTER_ID no Cluster id prefix (e.g. cluster-a). Omit for single-cluster setups.
AGBX_IMAGE_MAP no Path to a <task-name> <image> map file (one per line; = also accepted). If a task matches, that image is used verbatim. See Image selection.
AGBX_IMAGE_PREFIX no Mirror prefix applied to the task's docker_image (e.g. registry.internal/agent-sandbox). docker.io/ is stripped first. Not applied to AGBX_IMAGE_MAP values.
AGBX_IMAGE_TAG no Override the tag of the task's docker_image after rewriting. Not applied to AGBX_IMAGE_MAP values.
AGBX_HTTPS no true/false for the data-plane scheme (default true).
AGBX_STARTUP_TIMEOUT no Sandbox startup timeout, seconds (default 300).
AGBX_READY_TIMEOUT no Cold-image readiness ceiling, seconds (default 600). Large images (e.g. SWE-bench) may need more.

e2b SDK ≥ 2.24: newer e2b SDKs reject non-e2b_ API keys client-side. Use agent-sandbox-e2b >= 0.0.4, whose patch_e2b() neutralizes that check so agbx_ keys work (needed when running on harbor >= 0.13, which pulls a newer e2b).

Image selection

The image for each task is chosen in this order:

  1. AGBX_IMAGE_MAP entry — if the file maps the task name (Harbor's environment_name, i.e. the task / instance id) to an image, that image is used verbatim. This is how you run datasets whose task.toml has no docker_image (e.g. SWE-bench): pre-build / mirror the images once, list them here.

    # <task-name>  <image-ref>
    astropy__astropy-7606  registry.internal/agentbox/swebench/sweb.eval.x86_64.astropy_1776_astropy-7606:260328
    django__django-11265   registry.internal/agentbox/swebench/sweb.eval.x86_64.django_1776_django-11265:260328
    
  2. task.toml docker_image — if there's no map entry but the task sets [environment] docker_image (e.g. Terminal-Bench), that image is used, after optional AGBX_IMAGE_PREFIX / AGBX_IMAGE_TAG rewriting.

  3. Otherwise the task is rejected. This environment only runs pre-built images — it does not build images from a Dockerfile and does not mutate a running sandbox. Datasets that ship a Dockerfile (with extra RUN layers) must be built/mirrored ahead of time and listed in AGBX_IMAGE_MAP.

Example: SWE-bench (Dockerfile-based dataset)

# 1. Pre-build the images the dataset's Dockerfile would produce (base + your overlay),
#    push them to your registry, and write a map file:
#       astropy__astropy-7606  registry.internal/.../sweb.eval.x86_64.astropy_1776_astropy-7606:<tag>
#       ...
# 2. Point the plugin at it and run:
harbor run \
  -d swebench-verified@1.0 \
  -a oracle \
  --environment-import-path agent_sandbox_harbor:AgentSandboxEnvironment \
  --env-file swebench.env          # contains AGBX_IMAGE_MAP=swebench_image_map.txt

How it works

AgentSandboxEnvironment subclasses Harbor's E2BEnvironment and overrides three methods:

  • _does_template_exist → always returns True
  • _create_template → no-op
  • _create_sandbox → calls AsyncSandbox.create(template="cluster::pool//image", ...)

__init__ calls super().__init__() first, so Harbor's stock Dockerfile parsing still runs (and sets self._workdir from the image's WORKDIR). The constructor then resolves the image (see Image selection) and overrides self._template_name with the Agent Sandbox pool shorthand cluster::pool//image.

At module import, patch_e2b() from agent-sandbox-e2b redirects the e2b SDK to your Agent Sandbox endpoints.

See INTEGRATION.md for full design notes, the --environment-import-path mechanism explanation, and operational guidance.

Compatibility

Each release build is tested against the latest published versions of harbor and e2b. The pinned upper bound in [project.optional-dependencies] is updated automatically by the release CI to reflect the highest verified harbor version.

License

Apache 2.0.

Pre-provisioned environments and scaling groups

Provision the SandboxEnv and its member SandboxPools before running Harbor. The plugin only creates sandboxes from these resources; it does not create or resize environments, groups, or pools. Use the actual group names reported for the environment. Names are case-sensitive and are never synthesized from CPU or memory values.

There are two routing values: the E2B template is <cluster>::<env>//<image>, while create metadata agentbox.scitix.ai/scaling-group carries the selected group. The server only selects members of that Env in that group. An unavailable or nonexistent group fails instead of falling back to another member. This requires a server version that supports the reserved scaling-group metadata key; do not point the group mode at a direct member-pool name that bypasses Env routing.

For one fixed pre-provisioned group:

export AGBX_ENV_NAME=benchmark-env
export AGBX_SCALING_GROUP=balanced-workers

Without profiles this explicitly pins placement, but the plugin has no capacity inventory to validate. The operator must confirm the fixed group's real resource shape covers every task. GPU tasks always require profiles. To validate resource requirements and choose a group automatically, set AGBX_POOL_PROFILES to:

[
  {"env":"benchmark-env","scaling_group":"small-workers","cpus":1,"memory_mb":2048,"storage_mb":16384,"gpus":0},
  {"env":"benchmark-env","scaling_group":"balanced-workers","cpus":4,"memory_mb":8192,"storage_mb":16384,"gpus":0},
  {"env":"accelerator-env","scaling_group":"single-gpu","cpus":8,"memory_mb":32768,"storage_mb":65536,"gpus":1,"gpu_type":"H100"}
]

A profile may omit env when AGBX_ENV_NAME (or its legacy alias) supplies the default. An explicit profile env takes precedence over that default. Every Env-style profile must specify scaling_group, preventing capacity selection from being lost when the request reaches a multi-group Env. CPU, memory, and storage are compared with Harbor's effective task values, including resource overrides; missing declared capacity counts as zero. GPU count must match exactly, and a requested GPU model must be allowed. CPU-only tasks do not select GPU groups. Among fitting profiles selection is deterministic: CPU, memory, storage, Env, then group name in ascending order.

If AGBX_SCALING_GROUP is also set, it filters profiles by that exact group before selecting a resource fit. A too-small fixed group or absent group raises an error before AsyncSandbox.create; neither a larger group nor the legacy default target is silently substituted. Profiles are declarations that must match provisioned resources, not live capacity discovery. Invalid/empty profiles and non-finite or negative capacities fail closed.

Legacy {"pool":"target","cpus":...,"memory_mb":...,"storage_mb":...} profiles remain supported and retain their historical template-target meaning. They omit group metadata and therefore do not guarantee group-level placement inside a multi-group Env. Migrate them to env plus scaling_group for that purpose. If a profile contains both env and pool, they must agree.

Compose uses the same selection rules with AGBX_DIND_ENV_NAME, AGBX_DIND_POOL_PROFILES, and AGBX_DIND_SCALING_GROUP; it never inherits an ordinary task's group accidentally. Its template names the DinD Env without a task-image override, and its group applies to the outer sandbox, not individual Compose services.

GPU and separate verifier pools

The integration has been tested with Harbor 0.25.0 and 0.26.0. Configure AGBX_POOL_PROFILES with a JSON array describing already provisioned pools. The plugin chooses an exact GPU count and an allowed GPU model, with enough CPU, memory and storage. A GPU task without a matching profile fails before a sandbox is created; it must never fall back to a CPU pool. For example:

[
  {"pool":"cpu-pool","cpus":8,"memory_mb":16384,"storage_mb":30720,"gpus":0},
  {"pool":"gpu-pool","cpus":8,"memory_mb":16384,"storage_mb":30720,"gpus":1,"gpu_type":"H100"}
]

These are capacity declarations, not a provisioning API. Verify them against actual pool resources before running. Separate verifier environments select their own profiles and use their own pinned docker_image. Prefer the official prebuilt task bundle, which contains both image references, over a task-name image override that could incorrectly select the agent image for its verifier. Task startup environment variables and network policy are forwarded to E2B.

GPU routing unit tests do not establish live API access or full dataset support. Compose tasks use a separate pre-provisioned DinD pool; see the setup below.

Terminal-Bench 4: local Harbor, remote sandboxes

This section describes the GPU/Compose integration under development in this source tree. Do not assume an older PyPI release contains it. Check release notes for the verified plugin and Harbor versions before installing.

Install Python 3.12+ and the compatible Harbor/E2B extras. During development:

uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python 'harbor[e2b]==0.25.0' -e .

Provision isolated pools before running: a normal sandbox pool for CPU tasks, a GPU pool with nvidia.com/gpu requests/limits and appropriate node placement, and a DinD pool with a working Docker daemon and Compose v2. The GPU task image must also be compatible with the node driver. A CUDA image alone does not grant device access. Size the pools and their storage from the task requirements, including sidecars and the separate verifier. The example capacities above are for a small subset, not a capacity plan for the whole benchmark.

Set credentials privately in the local process. Copy the API URL and data-plane domain from your environment's connection documentation. Never commit keys:

# E2B_API_KEY must already be set through your private credential configuration.
export E2B_API_URL=https://sandbox.example.com/api/e2b
export E2B_DOMAIN=sandbox.example.com/api/data
export AGBX_CLUSTER_ID=example-cluster
export AGBX_POOL_PROFILES="$PWD/pools.json"
export AGBX_DIND_POOL_PROFILES="$PWD/dind-pools.json"
export AGBX_IMAGE_PREFIX=registry.example.com/benchmarks
export AGBX_STARTUP_TIMEOUT=900
export AGBX_READY_TIMEOUT=900

Both profile files use the same capacity schema. DinD profiles refer to pools whose default image starts dockerd. For a single known DinD environment, AGBX_DIND_POOL_NAME=dind-env can be used instead; the operator must ensure it has enough capacity. Every Compose trial claims a separate sandbox, so identical Compose project names do not collide across trials. Inner containers, networks and volumes are removed at stop; the outer sandbox is then released.

Use the official prebuilt release bundle, not the source tasks whose Dockerfiles still require builds. Terminal-Bench release assets include the bundle and a prebuilt-images.json manifest. Mirror every listed image using a manifest-preserving registry copier; verify target digests against that manifest and retain tags so registry garbage collection cannot remove untagged content. The prefix is applied to agent, verifier, and literal Compose service image references. Leave AGBX_IMAGE_TAG unset to preserve the official pins.

Run one task first, then the whole local task directory:

.venv/bin/harbor run -p ./tasks/math-eval-grader -a oracle -n 1 \
  --environment-import-path agent_sandbox_harbor:AgentSandboxEnvironment
.venv/bin/harbor run -p ./tasks -a oracle -n 1 \
  --environment-import-path agent_sandbox_harbor:AgentSandboxEnvironment

The controller and result files stay local. Commands, model inference and Docker Compose services run in the remote sandboxes. oracle exercises the published reference solutions; it does not measure an LLM agent's score. Inspect each trial's exception, verifier result and reward. Successful sandbox creation, image pulls, or process exit codes alone do not establish a passing benchmark.

Compose mode requires prebuilt service images. force_build and additional Compose overlays are not accepted by this adapter. Preserve authored service networks, health checks, dependencies and bind mounts in the release bundle. Per-service exec/download/stop operations use Harbor's DinD operations layer.

For release procedure and compatibility checks, see PUBLISHING.md.

CPU library thread budgets

CPU quotas do not necessarily restrict the host core count visible inside a container. OpenMP/BLAS libraries may otherwise create too many workers for a small pool. Set an explicit budget when needed:

export AGBX_CPU_THREADS=2

This sets OMP_NUM_THREADS, MKL_NUM_THREADS, and OPENBLAS_NUM_THREADS only when the task or persistent environment has not already set each variable. It is opt-in and must be recorded with benchmark results. It does not change task code, verifier thresholds, or timeouts.

Validation scope

The development implementation has exercised local Harbor 0.25.0 / E2B 2.63.0 against Terminal-Bench 4.0.0 reference solutions, including GPU tasks and all 11 Compose tasks. Of 66 tasks, 65 have completed with reward 1 in the validation setup. This is an infrastructure integration check, not an LLM leaderboard run. Two adaptations must be reported alongside that count:

  • cad-model: the agent environment preinstalls ocp_gordon==0.1.17 with build123d==0.10.0, avoiding an incompatible unbounded dependency resolution. The reference solution and verifier are unchanged.
  • vpp-loss-divergence: AGBX_CPU_THREADS=2, with the original verifier timeout.

The jax-speedrun-gpu task has not been validated in that setup because its 1000 GiB storage requirement exceeds the available node ephemeral capacity. Do not silently reduce the task requirement or report this task as passing. A deployment must also provide the task's CPU, memory, accelerator and storage capacity; profiles are declarations, not capacity reservations.

Some release archives include AppleDouble ._* metadata alongside real task files. Preserve the original archive; when preparing a Linux working copy, identify AppleDouble files by their header and exclude only verified metadata. Record the paths/hashes removed. Do not indiscriminately delete task files or change reference solutions to accommodate archive metadata.

Mirror inventory must include third-party Compose references as well as the release's prebuilt-image manifest. For this dataset, Kafka and Redis are direct runtime dependencies outside that manifest.

Terminal-Bench 2.1 / Harbor 0.26 compatibility

The local development plugin was tested with Harbor 0.26.0 and E2B 2.64.0, using Terminal-Bench 2.1 commit 12b7565679ce5006230b820666353673a133db93. Harbor 0.26 uses its public prebuilt_template_id startup path; the plugin retains upstream runtime preparation and lease renewal and supports the renamed sandbox timeout field. Unit tests also cover Harbor 0.25 compatibility.

The revised Golden suite has obtained reward 1 with no exception for all 89 distinct tasks across batches and targeted retries. The final qemu-alpine-ssh reference-solution synchronization correction passed three independent trials. This is 89/89 cumulative coverage of the disclosed adapted suite, not a single concurrent clean run, and not an unmodified official-oracle result.

The validation uses disclosed environment adaptations, not just original images:

  • build-cython-ext: a derived image supplies the compatible planarity 0.6 dependency; the original solution and verifier remain unchanged.
  • mcmc-sampling-stan: a derived image adds required build dependencies and records the dependency-upgrade policy used with the original solution.
  • protein-assembly: a derived image snapshots audited external RCSB input records and restores missing API metadata through a documented compatibility adapter. It contains scientific input data, not generated task answers.
  • QEMU tasks use documented runtime-image adaptations. For qemu-alpine-ssh, an additional explicit oracle patch waits for real console completion output rather than echoed command text. This is an explicit solution modification, so its successful result cannot be called unmodified official Golden.
  • custom-memory-heap-crash: a separate sandbox pool grants SYS_RESOURCE so the reference solution can genuinely raise its own core-dump limit. The envd memguard dependency otherwise initializes that inherited hard limit to zero. This capability is not enabled for the ordinary benchmark pool.

Original scoring assertions and thresholds are retained. The delivery audit must include every image recipe/digest, runtime setting, oracle patch and retry; do not pool diagnostic/modified-solution results into official-unmodified counts. The early 12-task smoke run, dependency-control experiments, and subsequent full batches remain separate evidence. build-pmars passes using the official 2.1 solution, which removes obsolete Debian package pins. A cluster mirror must have every rendered digest: a valid client map is insufficient if registry routing rewrites its host to a mirror missing that manifest.

These are reference-solution infrastructure checks, not LLM performance results. These changes have not been published to PyPI. The local 0.0.14 distribution records this validation scope; its delivery checksum identifies the exact build. Pinning images does not freeze all packages or services used at task runtime. Harbor 0.26's E2B backend kills sandboxes at stop even with --no-delete; collect diagnostics before stop.

Metadata

Release files for agent-sandbox-harbor 0.0.14

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for agent-sandbox-harbor 0.0.14
File Size Uploaded Provenance Actions
agent_sandbox_harbor-0.0.14.tar.gz 30.9 kB With provenance Details for agent_sandbox_harbor-0.0.14.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for agent-sandbox-harbor 0.0.14
File Interpreter ABI Platform Actions
agent_sandbox_harbor-0.0.14-py3-none-any.whl Python 3 none any Details for agent_sandbox_harbor-0.0.14-py3-none-any.whl

Total release size: 51.5 kB

Release files / agent_sandbox_harbor-0.0.14.tar.gz

Download URL agent_sandbox_harbor-0.0.14.tar.gz
Size 30.9 kB
Tags Source
SHA-256 checksum
How to use checksums
ac0d021f417511a302d934a583dc7a14e7ffeaa7a24c6bc1dd57601164471bdf
BLAKE2b-256 checksum
How to use checksums
5e14804bbaf96160bf2eea35bda09bd1bf702195047d126c1b0f7214dbefc7c0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / agent_sandbox_harbor-0.0.14-py3-none-any.whl

Download URL agent_sandbox_harbor-0.0.14-py3-none-any.whl
Size 20.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
45bc36dc2b5857a76e86f7c8fa11617a3bcf28d0d9cc7482cb2a794ea9f5b750
BLAKE2b-256 checksum
How to use checksums
7c606dc843aa5cf3c08bcf51062243395f16ba353921b72c39d2c7dad1a27eac
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Security for agent-sandbox-harbor 0.0.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for agent_sandbox_harbor-0.0.14.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page