Harbor Rewardkit

The Harbor Rewardkit is a lightweight package to define and run verifiers. Rewardkit is designed to be used with the Harbor task format but you can use it on its own.

Installation

uv tool install harbor-rewardkit

Example: Programmatic criteria

# tests/check.py
from rewardkit import criteria

criteria.file_exists("output.txt")
criteria.file_contains("output.txt", "hello")

Example: scoring config for a dimension

Drop a reward.toml next to the checks to control how they score. [scoring] gates the group instead of averaging it, and weight sets its share of the dimension when the directory also holds judge tomls:

# tests/structure/reward.toml
weight = 2.0

[scoring]
aggregation = "all_pass"

Directories can be nested into larger groups. A non-root directory may use one [[reward]] table to aggregate its local checks, judges, and immediate child directories. Child directories default to equal weight; override them with a short inline map:

# tests/correctness/reward.toml
[[reward]]
aggregation = "weighted_mean"
weights = { files = 2.0, behavior = 1.0 }

At the tests root, named aggregations use the same syntax:

[[reward]]
name = "reward"
aggregation = "weighted_mean"
weights = { correctness = 2.0, quality = 1.0 }

Root-level judge TOMLs also remain valid when dimension directories exist. Each becomes a top-level score named after its filename stem and may be included in the root aggregation.

Example: LLM judge

# tests/quality.toml
[judge]
judge = "anthropic/claude-sonnet-4-6"
files = ["/app/main.py"]

[[criterion]]
description = "Is the code correct?"
type = "binary"

Example: Agent judge with an MCP server

Each [[judge.mcp_servers]] entry matches a Harbor task's [[environment.mcp_servers]]. Per-server allowed_tools lists the tools the judge may call; omit it to allow all of the server's tools. Codex does not support sse servers.

# tests/quality.toml
[judge]
judge = "claude-code"

[[judge.mcp_servers]]
name = "playwright"
transport = "stdio"
command = "npx"
args = ["@playwright/mcp@latest", "--headless", "--isolated"]
allowed_tools = ["navigate", "click"]

[[criterion]]
description = "Does the rendered page match the spec?"
type = "binary"

Usage

Add rewardkit to your test.sh file:

# tests/test.sh
uvx harbor-rewardkit@0.1 /tests

See the documentation and a full working example.