This release has been yanked by its maintainers, and will be ignored by installers, except when explicitly specified.
Consider using release 0.23.0 instead.
Reason given by maintainers: Accidental nightly release during testing.
Harbor
Harbor is a framework from the creators of Terminal-Bench for evaluating and optimizing agents and language models. You can use Harbor to:
- Evaluate arbitrary agents like Claude Code, OpenHands, Codex CLI, and more.
- Build and share your own benchmarks and environments.
- Conduct experiments in thousands of environments in parallel through providers like Daytona, Modal, LangSmith, Blaxel, and Novita Sandbox.
- Generate rollouts for RL optimization.
Check out the Harbor Cookbook for end-to-end examples and guides.
Installation
uv tool install harbor
or
pip install harbor
Example: Running Terminal-Bench-2.0
Harbor is the official harness for Terminal-Bench-2.0:
export ANTHROPIC_API_KEY=<YOUR-KEY>
harbor run --dataset terminal-bench@2.0 \
--agent claude-code \
--model anthropic/claude-opus-4-1 \
--n-concurrent 4
This will launch the benchmark locally using Docker. To run it on a cloud provider (like Daytona) pass the --env flag as below:
export ANTHROPIC_API_KEY=<YOUR-KEY>
export DAYTONA_API_KEY=<YOUR-KEY>
harbor run --dataset terminal-bench@2.0 \
--agent claude-code \
--model anthropic/claude-opus-4-1 \
--n-concurrent 100 \
--env daytona
To see all supported agents, and other options run:
harbor run --help
To explore all supported third party benchmarks (like SWE-Bench and Aider Polyglot) run:
harbor datasets list
To evaluate an agent and model one of these datasets, you can use the following command:
harbor run -d "<dataset@version>" -m "<model>" -a "<agent>"
Citation
If you use Harbor in academic work, please cite it using the “Cite this repository” button on GitHub or the following BibTeX entry:
@software{Harbor_Framework,
author = {{Harbor Framework Team}},
title = {{Harbor: A framework for evaluating and optimizing agents and models in container environments}},
year = {2026},
version = {v0.16.1},
doi = {10.5281/zenodo.20953922},
url = {https://doi.org/10.5281/zenodo.20953922}
}
The DOI above is the concept DOI, which always resolves to the latest release and aggregates citations across all versions. To cite a specific version instead, use that version's DOI from the Zenodo record.
Release files for harbor 0.19.0.dev202607091635
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| harbor-0.19.0.dev202607091635.tar.gz | 1.5 MB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| harbor-0.19.0.dev202607091635-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 3.2 MB
Release files / harbor-0.19.0.dev202607091635.tar.gz
| Download URL | harbor-0.19.0.dev202607091635.tar.gz |
|---|---|
| Size | 1.5 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
0faa789416271f28cd6cd859a563751d8a8ddace48e5646079262c10f8a3680e
|
|
BLAKE2b-256 checksum How to use checksums |
5abb132a7ba8abbd69b08a15608959ef18e9d7f71da9e423c431562b13c624a6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.11.28 {"installer":{"name":"uv","version":"0.11.28","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / harbor-0.19.0.dev202607091635-py3-none-any.whl
| Download URL | harbor-0.19.0.dev202607091635-py3-none-any.whl |
|---|---|
| Size | 1.7 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b893fc8c84ced079ef1c1ac2a08b0a0834013f78a790cd4fedcf15662b370c2d
|
|
BLAKE2b-256 checksum How to use checksums |
3e202eaaa382a7ea88866c094d2aaf2ae2db66551ce73e23b265a134265a0350
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.11.28 {"installer":{"name":"uv","version":"0.11.28","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
