rlm-harness

A clean, reusable harness for building any task on top of DSPy's Recursive Language Model module (dspy.RLM).

RLMs (Zhang & Khattab, MIT, arXiv:2512.24601) let a model explore unbounded context by treating it as a variable in a sandboxed Python REPL and recursively calling sub-LLMs over it. DSPy's dspy.RLM is the first-party implementation (Khattab co-authored both DSPy and the RLM paper): it works with existing Signatures and is optimizer-compatible (GEPA/MIPRO). This kit distills the boilerplate around it into one small, opinionated layer.

rlm-harness is domain-agnostic: anything dspy.RLM can do fits: multi-hop "deep research", an RSS-digest agent that posts to a webhook, structured extraction, detection authoring, you name it. Security happens to be the author's own first use of it, but it isn't the kit's scope.

Why this exists

Using dspy.RLM directly leaves you re-writing the same plumbing for every task: model/sub-model config, a retry+validation loop, a sandbox choice, observability. rlm-harness makes a task a declaration:

from rlm_harness import RLMConfig, RLMTask, configure
from rlm_harness.tools import make_schema_validator
from pydantic import BaseModel

class Article(BaseModel):
    title: str
    summary: str

class Summarize(RLMTask):
    signature = "document: str -> article: Article"
    output_field = "article"
    output_model = Article
    instructions = "Read the document and produce a title and a one-paragraph summary."
    tools = [make_schema_validator(Article)]

configure(RLMConfig.from_env())
article = Summarize().run(document=long_text)   # validated Article

The retry loop, pydantic validation, sandbox selection, and budget caps are all inherited.

Installation

pip install rlm-harness
# or with uv:
uv add rlm-harness

rlm-harness needs Python ≥ 3.11 and pulls in dspy + pydantic. Everything else is an opt-in extra: [observe] (Langfuse/OpenInference tracing), [mcp] (the MCP client bridge), [jsonschema] (make_json_schema_validator), [grep] (a real wall-clock timeout on make_grep_files_tool's LM-supplied pattern), [gitignore] (.gitignore-aware walking in list_candidate_paths), and [subscription]: run the planner and/or sub-LM on a Claude Pro/Max login instead of an API key (pip install "rlm-harness[subscription]" → rlm_harness.ClaudeAgentLM, injected via configure(main_lm=…)). A live dspy.RLM run additionally needs model credentials (see the guide's Configuration) and a Deno sandbox: the logic and tests run without either. dspy 3.4.0 requires Deno >=2.4.5,<3.0.0: brew install deno, or let dspy manage it with pip install "dspy[deno]".

What's in the box

• Tasks as declarations. Subclass RLMTask: the retry+validation loop, sandbox selection, budget caps, and observability are inherited.
• The whole trajectory, recorded. TraceRecorder writes main steps, every sub-LM call, and every tool call into one append-only JSONL stream: replayable and exportable as SFT/RL datasets (reward-free: scoring belongs to your trainer). Offline readers get the generic half computed for them: run facts, rubric facts, and utilization metrics, including which tool calls produced nothing usable, and what they cost.
• The recursion seat, interceptable. Every sub-LM escalation is traced as a sub_call automatically: no wrapper needed. intercept_sub_lm adds a deterministic validate/post-process pipeline on top; model_as_tool lets the main LM choose to consult another named model, in the trajectory.
• Tools, the base/wrap way. Pydantic/JSON-Schema validators, an SSRF-guarded fetch_url, provider-agnostic web search, the generic model-as-tool core, a run_command seam over your isolated runner, an MCP client bridge, and skills-as-tools progressive disclosure.
• A bounded local directory, no shell. Read, write, edit, and regex-search files under a root you scope, plus safe archive extraction and git clone. Every path resolves inside that root (no subprocess, no rg on PATH), and verify_quote checks a model's citation against the bytes it claims to quote.
• Delegate to another harness, or be one. make_harness_tool wraps a downstream rlm-harness harness as a tool: the parent records one tool_call plus a link to the child, while the child runs its own full RLM loop over the long text and owns its own trace. serve_harness is the other end of the same wire. The kit ships no transport and names no harness: the identity lives in your runtime config.
• Sandboxed by default. The pyodide/deno interpreter; the local interpreter is refused unless explicitly opted into; an opt-in Docker container interpreter for when the REPL itself needs real subprocesses.
• Offline-testable. rlm_harness.testing drives the real dspy.RLM forward loop with no model, no Deno, no network.

Documentation: the guide

The deep documentation lives in rlm_harness/README.md:

• Layout: what each module owns.
• RLM as harness engineering: the sub-LM hook + trajectory tracing.
• Sub-LM vs. tool: which model goes where; the choice decides what your RL data records.
• Skills, MCP tools, running local commands, and run_in_subprocess: the tool surfaces and the isolated-subprocess primitive under them.
• Reading and searching local files, writing and editing them, and extracting archives: a bounded directory, no shell.
• The container interpreter, sandbox turn timeout + cancellation, and what bounds what: the execution environment and its deadlines.
• Grounded completeness and judgement-only SUBMIT: the rollout conventions.
• Rubric facts and trace utilization metrics: the generic half of reading a finished run, computed for you and reward-free.
• Reading a trace: the ordering rules: step_id is WRITE order, and step_id adjacency means nothing.
• Building a consumer: the six-step extension contract.
• Configuration: every env var, adapter selection, model naming.
• Testing the forward path offline: the scripted offline harness.

Built with rlm-harness

Real projects using rlm-harness as their RLM scaffold:

• Penumbra: a personal knowledge hub that runs on your own machine. Capture sources of any kind (text, web pages, PDFs including scanned ones, YouTube captions) into one stream, gather them into Orbits, ask questions grounded in them, and get back a citation you can open and check against the original. Ships as a desktop app, with a Trajectory drawer over the run that produced each answer.
• ctx-distillery: distils an AI coding agent's session transcripts and memory store into a judgement-only distillation plan: what to prune, cross-reference, or promote into durable memory or a reusable Skill. It proposes; it writes nothing.
• cve-reverser: reverses publicly disclosed CVEs from their patches into local-lab PoCs and Nuclei detection templates. A traced, trainable RLM harness.
• diff-sentry: classifies GitHub changes (PRs, issues, pushes) for malicious intent. The diff is read as untrusted data in the sandboxed REPL, emitting evidence-backed benign / suspicious / malicious verdicts into a SIEM.
• toolscout: an ATLAS-style rollout harness, a small planner progressively discovers a large MCP toolspace and computes over tool results as code, emitting reward-free trajectories + per-criterion facts for a downstream trainer.

Built something on rlm-harness? Open a PR to add it here.

Security note: the sandbox is the boundary

RLM executes model-written code. When that code processes untrusted scraped content, the interpreter choice is your attack surface. The default (pyodide/deno) is the sandboxed DSPy interpreter. The local interpreter runs code on the host and is refused unless you set allow_insecure_sandbox=True / RLM_ALLOW_INSECURE_SANDBOX=1. Don't.

The default sandbox is built by the kit (not handed straight to dspy) so it can pre-bind the JSON literals true/false/null to True/False/None in the REPL namespace: a JSON-trained instruct model otherwise writes SUBMIT({"ok": true}) and the REPL raises NameError: name 'true' is not defined, which the model tends to retry verbatim. Isolation is unchanged; RLMTask owns the teardown.

Develop

uv sync --group dev
uv run pytest          # logic tests (no live LLM needed)

The optional extras carry their own tests, which SKIP when the extra is absent. What CI runs is uv run --group dev --extra mcp --extra grep --extra gitignore python -m pytest -q, plus uvx ruff check . as a separate gate.

Tests cover config parsing, the retry/validation engine, the sandbox guard, the tools, the sub-LM-hook/trace/replay/dataset layer, and a real-dspy.RLM construction check (dspy-bearing tests use DummyLM or skip if dspy is absent). A live run additionally needs real credentials and a Deno sandbox (brew install deno, or pip install "dspy[deno]" for dspy's managed binary; it requires Deno >=2.4.5,<3.0.0); examples/mini_run.py shows it. To drive the real forward loop offline (no model, no Deno), see the guide's Testing the forward path offline. See CLAUDE.md for invariants when modifying the kit.

Status

Released versions, with what changed in each, are on the Releases page and in CHANGELOG.md. This section used to restate the current one and fell five versions behind, so it no longer tries.

What is worth saying here is the part that does not change with a version number.

1.0.0 means the public surface is a contract: __init__.__all__, the rlm-harness/trace/v1 wire format, and RLMTask's declaration fields are frozen under SemVer and pinned by tests/test_contract.py. Additions ship in a minor release; a rename or removal ships with an alias and a DeprecationWarning first, and the removal itself waits for the next major. The trace format carries its own version and evolves additive-only within v1. A _-prefixed name or module internal is not part of that promise.

Next: enable optimize.compile_task against a labelled trainset to actually GEPA-compile tasks (currently a documented stub).

License

MIT © Boik Su (@boik_su). See LICENSE.