harbor-aws 0.2.1

AWS EKS/Fargate execution backend for Harbor benchmarks

harbor-aws

AWS EKS/Fargate execution backend for Harbor benchmarks.

• Low infrastructure overhead: One line AWS infrastructure creation and destroy.
• High-concurrency execution: Run Harbor benchmarks at max concurrency on AWS.
• Pay-on-demand execution: Ensure cost scales with benchmark demand.

Install

uv sync --extra cdk

Quick Start

# Deploy infrastructure (one-time, ~15 min)
uv run python -m harbor_aws deploy

# Run benchmarks
uv run harbor run -c job-config.yaml \
  -d terminal-bench@2.0 \
  -a terminus-2 \
  -m bedrock/converse/moonshotai.kimi-k2.5 \
  -n 89

# Clean up
uv run python -m harbor_aws stop      # delete pods, keep infra
uv run python -m harbor_aws destroy   # delete everything

Prerequisites: AWS account with admin access. Docker Hub login (docker login) recommended to avoid anonymous pull rate limits.

Scaling

Image pulls are capped at 50 concurrent operations by default to avoid Docker Hub rate limiting. For higher sustained concurrency, configure Amazon ECR pull-through cache for Docker Hub images.

ECR pull-through cache setup

During deploy, you'll be prompted to provide Docker Hub credentials. If provided, the deploy will automatically create the Secrets Manager secret and ECR cache rule.

To set it up manually instead:

aws secretsmanager create-secret \
  --name ecr-pullthroughcache/docker-hub \
  --secret-string '{"username":"YOUR_DOCKERHUB_USER","accessToken":"YOUR_ACCESS_TOKEN"}' \
  --region us-east-1

Then enable in job config:

environment:
  import_path: "harbor_aws.adapter:AWSEnvironment"
  kwargs:
    stack_name: harbor-aws
    region: us-east-1
    ecr_cache: true

Validation

Benchmarks reproduced from the Kimi K2.5 technical report using Kimi K2.5 on Amazon Bedrock with terminus-2.

Benchmark	Official	harbor-aws
SWE-bench Verified	76.8%	71.5%
Terminal-Bench 2.0	50.8%	43.8%
GPQA-Diamond	87.6%	79.8%
LiveCodeBench v6	85.0%	88.6%

Score gaps are expected — official results used Kimi's internal agent for some benchmarks, while we use terminus-2 throughout.

Documentation

• System Architecture & Design Principles — architecture overview, tradeoffs, and design rationale

Development

uv sync --extra dev --extra cdk
uv run ruff check src/
uv run mypy src/

License

Apache License 2.0 — see LICENSE.