This release is a pre-release and may not be stable for production use.
Harbor
Harbor is a framework from the creators of Terminal-Bench for evaluating and optimizing agents and language models. You can use Harbor to:
- Evaluate arbitrary agents like Claude Code, OpenHands, Codex CLI, and more.
- Build and share your own benchmarks and environments.
- Conduct experiments in thousands of environments in parallel through providers like Daytona, Modal, LangSmith, Blaxel, Novita Sandbox, Tensorlake, Runta, and Sail.
- Generate rollouts for RL optimization.
Check out the Harbor Cookbook for end-to-end examples and guides.
Installation
uv tool install harbor
or
pip install harbor
Example: Running Terminal-Bench-2.0
Harbor is the official harness for Terminal-Bench-2.0:
export ANTHROPIC_API_KEY=<YOUR-KEY>
harbor run --dataset terminal-bench@2.0 \
--agent claude-code \
--model anthropic/claude-opus-5-5 \
--n-concurrent 4
This will launch the benchmark locally using Docker. To run it on a cloud provider (like Daytona) pass the --env flag as below:
export ANTHROPIC_API_KEY=<YOUR-KEY>
export DAYTONA_API_KEY=<YOUR-KEY>
harbor run --dataset terminal-bench@2.0 \
--agent claude-code \
--model anthropic/claude-opus-5-5 \
--n-concurrent 100 \
--env daytona
To see all supported agents, and other options run:
harbor run --help
To explore all supported third party benchmarks (like SWE-Bench and Aider Polyglot) run:
harbor datasets list
To evaluate an agent and model one of these datasets, you can use the following command:
harbor run -d "<dataset@version>" -m "<model>" -a "<agent>"
Citation
If you use Harbor in academic work, please cite it using the “Cite this repository” button on GitHub or the following BibTeX entry:
@software{Harbor_Framework,
author = {{Harbor Framework Team}},
title = {{Harbor: A framework for evaluating and optimizing agents and models in container environments}},
year = {2026},
doi = {10.5281/zenodo.20953922},
url = {https://doi.org/10.5281/zenodo.20953922}
}
The DOI above is the concept DOI, which always resolves to the latest release and aggregates citations across all versions. To cite a specific version instead, use that version's DOI from the Zenodo record.
Release files for harbor 0.24.1.dev202610200418
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| harbor-0.24.1.dev202610200418.tar.gz | 2.2 MB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| harbor-0.24.1.dev202610200418-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 4.7 MB
Release files / harbor-0.24.1.dev202610200418.tar.gz
| Download URL | harbor-0.24.1.dev202610200418.tar.gz |
|---|---|
| Size | 2.2 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d51670bde9e2d130b6b4d856e2c19c344d2ba5e854d942926527f47c81034d0b
|
|
BLAKE2b-256 checksum How to use checksums |
a82f448fbbce498fafa97d4435e9fa8d9327f77814a54c5f32fd90d550d00042
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.3 {"installer":{"name":"uv","version":"0.13.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / harbor-0.24.1.dev202610200418-py3-none-any.whl
| Download URL | harbor-0.24.1.dev202610200418-py3-none-any.whl |
|---|---|
| Size | 2.5 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e6ccaf05e46de63787245432854fa707f0e368d76eb688f836da10d5af133495
|
|
BLAKE2b-256 checksum How to use checksums |
07020d81a4ccbfbc1343783a61a5d0b05eb609aaefebed9becedef57ee0b71f3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.3 {"installer":{"name":"uv","version":"0.13.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for harbor 0.24.1.dev202610200418
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
