Gandalf the Grader

Gandalf the Grader is an Agent-as-a-Judge verifier for reinforcement learning environments. Gandalf provides an evaluation score / reward signal, grading the final state of the environment (including text response from agents, state of the filesystem, and state backing MCP tools) against a set of natural-language criteria specified in a rubric.

Unlike LLM-as-a-Judge or simple workflows (e.g., serialize-then-judge), Gandalf is capable of verifying outputs that are complex files, such as Excel or PowerPoint deliverables, and checking them against sophisticated rubric criteria, like "sensitivity table is properly constructed using Excel Data Table functionality or equivalent formula array." Gandalf is able to do this because it grades rubric criteria by running an AI agent within the RL environment itself.

See BankerToolBench as a complete example of Gandalf being used as the verifier in a production RL environment. See rle-pkg as a reference RL environment runtime that includes integration with Gandalf.

Gandalf is agnostic to RL environment framework/runtime; we have been using it mainly with Harbor.

Installation

Gandalf is published on PyPI.

uv tool install gandalf-the-grader

For production use, we recommend that you pin a specific version of Gandalf, and furthermore use the [pinned] version to pin all transitive dependencies.

uv tool install 'gandalf-the-grader[pinned]==1.0.0'

Quick start

Create a grader config (grader.toml):

model = "gemini/gemini-2.5-flash"
sandbox_user = "sandbox"
instructions = "Build a web app that displays hello world."
rubric_path = "/tests/rubric.json"
workdir = "/home/agent/workspace"
trajectory_path = "/logs/agent/trajectory.json"
output_dir = "/logs/grader"

Create a rubric (rubric.json):

[
  {"criterion": "The file index.html exists in the workspace", "weight": 1.0},
  {"criterion": "The page displays 'Hello World'", "weight": 2.0}
]

Run the grader:

gandalf-the-grader --config /tests/grader.toml

Configuration

grader.toml

MCP servers can be configured as TOML array of tables:

[[mcp_servers]]
name = "magic-server"
transport = "stdio"
command = "/usr/bin/mcp-server"
args = ["--verbose"]

Custom Judge Prompt

By default, the grader uses a built-in prompt template to kick off each judge session. judge_prompt / judge_prompt_path let you replace it entirely with a custom Jinja2 template.

Note: This prompt is sent as the opening user message to the judge agent, not the LLM system prompt. The underlying agent framework (OpenHands) has its own immutable system message with coding and tool-use instructions that we never modify. Our prompt sits on top of that as the first user turn, setting up the grading task.

For most use cases, judge_guidance / judge_guidance_path is all you need: it injects extra instructions into the built-in prompt without replacing it. Fully overriding the judge prompt is an uncommon escape hatch for situations where the built-in prompt structure itself is unsuitable.

The template receives these variables:

Individual and batch modes use separate built-in templates. In a custom template, use {% if criterion is defined %} vs {% if criteria is defined %} if you need to distinguish modes. In batch mode, use loop.index0 for the criterion index (e.g., {% for c in criteria %}[{{ loop.index0 }}] {{ c }}{% endfor %}).

Rubric JSON

A JSON array of objects with criterion (string) and weight (float). Weights can be negative to penalise undesired outcomes:

[
  {"criterion": "The output file exists", "weight": 2.0},
  {"criterion": "The output contains correct totals", "weight": 3.0},
  {"criterion": "The agent used hardcoded values instead of computing", "weight": -1.0}
]

• Positive weight: adds to the raw score when the criterion's condition is met
• Negative weight: deducts from the raw score when the criterion's condition is met (the bad thing happened)
• The judge evaluates each criterion on its own merits; it never sees weights

Trajectory Format (ATIF)

The grader reads agent trajectories in Agent Trajectory Interchange Format (ATIF). An ATIF file is a JSON object with a steps array:

{
  "steps": [
    {"source": "user", "message": "Build a hello world web app"},
    {"source": "agent", "message": "I'll create the file now", "tool_calls": [...]},
    {"source": "agent", "message": "Done! I created index.html with a Hello World page."}
  ]
}

The grader extracts the final agent message (last "source": "agent" step with a non-empty message and no tool_calls) and passes it to the judge as context.

Environment Variables

Tracing / Observability

Gandalf builds on top of OpenHands, which has built-in OpenTelemetry tracing that automatically instruments LLM calls, tool executions, and agent steps. Set the OTEL_EXPORTER_OTLP_* variables above to export traces to any OTEL-compatible backend with no code changes required.

Example: Langfuse

# Encode your Langfuse keys
echo -n "pk-lf-...:sk-lf-..." | base64

# Export the variables
export OTEL_EXPORTER_OTLP_ENDPOINT=https://cloud.langfuse.com/api/public/otel/v1/traces
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic%20<base64-encoded-keys>"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf

Output

The grader writes to output_dir:

• reward.json: Reward file (e.g., {"reward": 0.75}) (always in [0, 1]). Only written when all criteria are successfully evaluated. If any criteria still have errors after retries, the grader writes info.json but skips reward.json and exits with code 1.
• info.json: Always written. Per-criterion results with met/not-met, reasoning, evidence, LLM usage, plus reward, raw_score, minimum_score, maximum_score, errored_criterion_count, and evaluated_criteria_pct.
• judge_trace_*.txt: stdout/stderr capture for each judge invocation. Naming varies by mode: judge_trace_{i}.txt (individual), judge_trace_batch.txt (batch), judge_trace_batch_split{i}.txt (batch with splits). Retries append a _retry{N} suffix.

The reward in reward.json is clip(0, 1, raw_score / sum_of_positive_weights), always in [0, 1]. info.json additionally includes raw_score (the raw sum of weights for met criteria, which can be negative) and minimum_score/maximum_score bounds for reference.

License

Copyright (c) Handshake. Released under the Apache-2.0 license. See LICENSE.txt for details.