Skip to main content

Harbor

Docs Cookbook DOI

Harbor is a framework from the creators of Terminal-Bench for evaluating and optimizing agents and language models. You can use Harbor to:

  • Evaluate arbitrary agents like Claude Code, OpenHands, Codex CLI, and more.
  • Build and share your own benchmarks and environments.
  • Conduct experiments in thousands of environments in parallel through providers like Daytona, Modal, LangSmith, Blaxel, Novita Sandbox, Tensorlake, Runta, and Sail.
  • Generate rollouts for RL optimization.

Check out the Harbor Cookbook for end-to-end examples and guides.

Installation

uv tool install harbor

or

pip install harbor

Example: Running Terminal-Bench-2.0

Harbor is the official harness for Terminal-Bench-2.0:

export ANTHROPIC_API_KEY=<YOUR-KEY> 
harbor run --dataset terminal-bench@2.0 \
   --agent claude-code \
   --model anthropic/claude-opus-5-5 \
   --n-concurrent 4 

This will launch the benchmark locally using Docker. To run it on a cloud provider (like Daytona) pass the --env flag as below:

export ANTHROPIC_API_KEY=<YOUR-KEY> 
export DAYTONA_API_KEY=<YOUR-KEY>
harbor run --dataset terminal-bench@2.0 \
   --agent claude-code \
   --model anthropic/claude-opus-5-5 \
   --n-concurrent 100 \
   --env daytona

To see all supported agents, and other options run:

harbor run --help

To explore all supported third party benchmarks (like SWE-Bench and Aider Polyglot) run:

harbor datasets list

To evaluate an agent and model one of these datasets, you can use the following command:

harbor run -d "<dataset@version>" -m "<model>" -a "<agent>"

Citation

If you use Harbor in academic work, please cite it using the “Cite this repository” button on GitHub or the following BibTeX entry:

@software{Harbor_Framework,
author = {{Harbor Framework Team}},
title = {{Harbor: A framework for evaluating and optimizing agents and models in container environments}},
year = {2026},
doi = {10.5281/zenodo.20953922},
url = {https://doi.org/10.5281/zenodo.20953922}
}

The DOI above is the concept DOI, which always resolves to the latest release and aggregates citations across all versions. To cite a specific version instead, use that version's DOI from the Zenodo record.

Metadata

Release files for harbor 0.27.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for harbor 0.27.0
File Size Uploaded Provenance Actions
harbor-0.27.0.tar.gz 2.6 MB Without provenance Details for harbor-0.27.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for harbor 0.27.0
File Interpreter ABI Platform Actions
harbor-0.27.0-py3-none-any.whl Python 3 none any Details for harbor-0.27.0-py3-none-any.whl

Total release size: 5.4 MB

Release files / harbor-0.27.0.tar.gz

Download URL harbor-0.27.0.tar.gz
Size 2.6 MB
Tags Source
SHA-256 checksum
How to use checksums
043f35a1fb8ef5f1dfac7e23c2b27c818d17b59f9782a3ef21e10844e2ab089e
BLAKE2b-256 checksum
How to use checksums
b224f1819360ac28b889ce9c3e0de3c3cd991271ea651f3f649dad0794491270
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / harbor-0.27.0-py3-none-any.whl

Download URL harbor-0.27.0-py3-none-any.whl
Size 2.8 MB
Tags Python 3
SHA-256 checksum
How to use checksums
79c4eea06e02229e36d85a584b447c907f936b52c02368ded9d0a1bb27bd3a00
BLAKE2b-256 checksum
How to use checksums
91472c2dffef319bfc42ba409689b1018eeda8ea6f664c3df37355443c4b7328
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for harbor 0.27.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.27.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page