Harbor Rewardkit

Rewardkit is a lightweight verifier package for Harbor tasks or standalone use.

Installation

uv tool install alpen

Programmatic verifier

# tests/checks.py
from rewardkit import criteria

criteria.file_exists("output.txt")
criteria.file_contains("output.txt", "hello")

Run it with:

uvx --from alpen rewardkit /tests

Rewardkit runs the criteria against /app and writes /logs/verifier/reward.json. With files directly under tests/, their scores are combined into one reward automatically.

Configure scoring

Each Python file that registers criteria is one equal-weighted component named after its filename stem. Use [scoring.<stem>] in reward.toml to change how the criteria within that file combine:

# tests/reward.toml
[scoring.checks]
aggregation = "all-pass"

Files that only provide imports or shared=True criterion definitions and register no checks are ignored.

Judge verifier

# tests/quality.toml
[judge]
judge = "anthropic/claude-opus-5-5"
files = ["/app/main.py"]

[[criterion]]
description = "Is the code correct?"
type = "binary"

judge is a LiteLLM model, an agent such as claude-code, or jev for JEV, a new type of language model from TypeSafe (install alpen[jev]).

Python files and judge TOMLs can coexist. A judge TOML is also a component named after its stem and has its own optional [scoring] section.

To change how these flat components combine, add a named reward aggregation:

# tests/reward.toml
[[reward]]
name = "reward"
aggregation = "weighted-mean"
weights = { checks = 2.0, quality = 1.0 }

Multi-reward verifier

Put scoring inputs in subdirectories to emit separate dimensions:

tests/
├── correctness/
│   └── checks.py
├── structure/
│   └── files.py
└── quality/
    └── judge.toml

This emits correctness, structure, and quality. Add a root tests/reward.toml to combine them into a main reward:

[[reward]]
name = "reward"
aggregation = "weighted-mean"
weights = { correctness = 2.0, structure = 1.0, quality = 1.0 }

Inside a dimension, one unnamed [[reward]] can set weights across local Python files, judge TOMLs, and child directories. Reference files by stem and child groups by directory name.

See the documentation for custom criteria, agent judges, the JEV judge, MCP servers, nested groups, and the complete config reference. A working Harbor task shows the full layout.