graded 1.0.5

Defensive verifier framework and helpers for Harbor evaluations

Graded

Graded is a library to make computing rewards simple, defensive, and structured for agent evaluations, particularly within Harbor environments. It provides tools to declare structured grading criteria, execute LLM judges with automatic tracing, and manage evaluation artifacts.

Installation

pip install graded

Or using uv:

uv pip install graded

Quick Start

Create an evaluation script (e.g. verify.py) to grade a task workspace:

from pathlib import Path
from graded import Evaluator

# Initialize the evaluator
ev = Evaluator(
    workspace="/workspace",
    output_path="/logs/verifier/reward.json",
    auto_save_artifacts=True
)

# 1. Declare a standard criterion (workspace parameter is optional)
@ev.criterion(name="has_output_file", weight=1.0)
def check_output() -> bool:
    return ev.file_exists("output.txt")

# 2. Declare a fatal criterion (takes workspace Path parameter to inspect files directly)
@ev.criterion(name="no_syntax_errors", weight=2.0, fatal=True)
def check_syntax(workspace: Path) -> bool:
    # Use workspace parameter to inspect the files on disk
    return (workspace / "src").is_dir()

# 3. Declare a fractional scoring criterion
@ev.criterion(name="test_pass_rate", weight=3.0)
def check_tests() -> float:
    return 0.8  # Returns a score between 0.0 and 1.0

if __name__ == "__main__":
    ev.run()

Core Features

1. Criteria Declarations (@ev.criterion)

Define check functions using the @ev.criterion decorator.

Check functions can optionally accept the workspace directory as a pathlib.Path parameter if they need to perform custom filesystem operations. If a function does not accept any arguments, it will be executed without the workspace parameter.

• name: Unique identifier for the criterion.
• weight: Relative weight of the score in the final weighted average calculation.
• fatal: If True, any score of 0.0 or False immediately short-circuits the final score to 0.0.
• Return Value: Must return a bool, int, or float.

Programmatic Registration (Without Decorators)

If you prefer to define functions normally, you can register them programmatically without using decorator syntax:

def check_output(workspace: Path) -> bool:
    return (workspace / "output.txt").is_file()

# Register directly
ev.criterion("has_output_file", weight=1.0)(check_output)

2. LLM Judge with Automatic Tracing

Integrate with instructor to run structured, schema-validated LLM grading prompts. Prompt, parameters, response schema, and LLM responses are automatically logged to traces.json.

from pydantic import BaseModel, Field

class Rubric(BaseModel):
    score: float = Field(description="Score between 0.0 and 1.0 based on correctness.")
    reasoning: str = Field(description="Detailed reasoning for the score.")

# In your criterion:
result = ev.llm_judge(
    model="google/gemini-3.5-flash",
    response_model=Rubric,
    system="You are a strict code correctness evaluator.",
    prompt="Compare the student's solution in code.py with the requirements...",
)

# The return value is fully type-hinted as an instance of your Rubric class
print(result.score)
print(result.reasoning)

3. File & Artifact Management

Access files and copy evaluation artifacts to the logs directory safely:

• ev.read_file(filename): Reads content as a string and auto-saves a copy to artifacts.
• ev.load_json(filename): Parses JSON file content and auto-saves a copy to artifacts.
• ev.save_file(filename, content): Saves arbitrary text to the artifacts directory.
• ev.save_dir(dirname): Copies an entire directory from the workspace to the artifacts directory.
• ev.load_trajectory(path): Loads and parses an agent's ATIF trajectory.json file.

Outputs

When ev.run() completes, the following files are written to the directory containing your configured output_path:

1. reward.json: Flat JSON dictionary containing the final calculated reward and individual scores.
2. reward.txt: Text file containing just the final reward float value.
3. traces.json: List of structured LLM calls made via ev.llm_judge.
4. metadata.json: Optional metadata.
5. artifacts/: Subfolder containing copy-back files preserved during the evaluation run.

Agent Skills

You can install the graded-verifier skill to teach your AI coding agents (such as Cursor or Claude Code) how to write robust graded verifiers:

npx skills add <github-username>/eval-helpers/.agents/skills/graded-verifier