onejudge

A Rust library that drives a simulated interaction and evaluation loop on top of oneharness: take a skill or agent, drive it through a multi-turn conversation with a simulated user, and score the resulting transcript with natural-language (judge) verdicts and tool-event queries.

It is the engine extracted from skilltest (see nickderobertis/skilltest#31). The layering:

oneharness  →  one harness invocation, one JSON report   (pure substrate)
onejudge    →  simulated interaction + judging loop        (this crate)
skilltest   →  test-framework surface: cases, evals-as-assertions, SDKs

Reach for onejudge when you want to "drive a harness through a simulated conversation and score the transcript" without skilltest's YAML / case framing.

Install

cargo add onejudge

Minimum supported Rust version: 1.82.

The onejudge CLI

The same engine that tests a skill can drive real work. onejudge run points a harness at a task and lets an LLM-driven simulated user supervise it — pushing back, asking for verification, re-prompting — until a done_when condition holds or max_turns is hit. Configured by YAML; the library API is unchanged and CLI deps (clap, a YAML parser) are opt-in behind the non-default cli feature.

Spin up a run in three steps:

cargo install onejudge --features cli   # or: install.sh (prebuilt archives)
onejudge init                           # scaffold onejudge.yaml + oneharness configs
onejudge run                            # reads ./onejudge.yaml, drives to completion

init shells out to oneharness init (needs oneharness 0.14.0+) to scaffold oneharness.toml (the agent side) and oneharness.judge.toml (the judge side), then writes a fully-commented loop-only onejudge.yaml. The fields that make a run yours are task (what to do), the system framing — a skill (a SKILL.md directory) and/or a system_prompt, both optional — and the user block (persona / done_when / max_turns — omit it for a single-turn run). After each nonterminal agent turn, one unified supervisor call either completes with a reason or supplies the exact next user message. It sees compact normalized tool summaries by default, never raw dumps; when needed it may inspect the agent-side recording with oneharness history show <session>-skill --project <worktree> --format text. Agent and judge harnesses run in that worktree, but only agent runs are automatically history-recorded. Harness and model selection lives in those oneharness.toml files, not onejudge.yaml. onejudge schema prints the annotated config, the single source of truth for every field.

Flags override the file (flags > file > defaults), so one config serves many tasks: onejudge run --task - < task.txt, --max-turns 8, --format json -o result.json.

Config

A run is a YAML file carrying only the loop's own concerns. The fields that make it yours — task, the system framing (skill and/or system_prompt), and the user block. Everything else has a default; omit user for a single-turn run. A minimal config:

system_prompt: You are a senior engineer. Complete the task and keep tests green.
# skill: ./skills/my-skill    # optional: a SKILL.md dir; its body is appended

task: Add a --version flag to the CLI.

user:                         # the simulated supervisor that drives the loop
  persona: A demanding tech lead. Do not accept "done" until you have verified it.
  done_when: the task is complete and all tests pass
  max_turns: 8

evals:                        # optional: score the finished transcript
  - criterion: the change is well-scoped and readable
    kind: numeric
    scale: [1, 5]
assessment: Identify useful follow-up work left out of scope.

The harness and model come from oneharness's own config (oneharness.toml for the agent, oneharness.judge.toml for the judge side) — onejudge init scaffolds them. More keys — provider (oneharness / command / split, with the oneharness judge_config path, and under split the judge side as a judges: list — an llmlint entry there makes a judge of one llmlint run over the worker's tree), session, boolean evals. onejudge init writes a fully-commented starter and onejudge schema prints the annotated field reference (the single source of truth); it is validated strictly (deny_unknown_fields) so a typo is a loud error.

Human output is the conversation + tool actions + completion status + eval verdicts; --format json emits the versioned Report. The exit code is 0 only when the task completed and every boolean eval passed, 1 if it hit max_turns or a boolean eval failed, 2 on a bad config. Add --stream to publish tool events on stdout as they occur, ahead of that same report (docs/streaming.md). Full docs: docs/cli.md.

Concepts

• Provider is the boundary — onejudge never talks to a model directly. Every model call goes through oneharness, and harness/model selection lives in oneharness's config files, not onejudge.
  • OneharnessProvider (default) shells out to the oneharness CLI (v0.14.0+): the agent side uses the discovered oneharness.toml, and the judge side uses a separate --config file (default oneharness.judge.toml). With stream: true it consumes the agent turn as it happens — NDJSON tool events, then a terminal report — instead of only when it ends (docs/streaming.md).
  • CommandProvider speaks a small JSON-lines protocol, for a custom backend or a deterministic test double.
  • LlmlintProvider is a judge whose whole verdict is one llmlint run over the worker's tree, met at the process boundary (nothing linked; llmlint installed on the host): failing rules send the worker llmlint's own report, a clean run passes it, and a run that could not complete is an error, never a verdict (docs/judges.md).
  • JudgePanel makes the judge side a list: every judge runs against the same worker turn at the same time, the panel waits for all of them, and the worker gets one combined, attributed message when any says the work is not done; each judge's decision is recorded on the report. A panel of one is byte-identical to a bare provider. See docs/judges.md.
  • SplitProvider composes two providers — one that runs the skill, one that judges and role-plays the user (a single provider or a JudgePanel; e.g. run the skill on one harness, judge on another).
• Engine runs a Conversation (a Skill, an initial input, and an optional SimulatedUser) into a Transcript, bounded by max_turns / done_when / the skill declaring itself done.
• Transcript carries each turn plus the normalized **ToolEvent**s the skill took, so the judge — and a ToolQuery — can reason over what the skill did, not just what it said.
• Report is onejudge's own versioned contract (SCHEMA_VERSION): a serializable bundle of the transcript, verdicts, optional free-text assessment, and usage that higher-level frameworks compose over and re-export. See docs/contract.md.
• SpawnHook is the seam for an in-process embedder: onejudge offers every process it is about to create so the embedder can place it in a group it owns — a POSIX process group, a Windows job object — and still reap the whole harness tree when a run is cancelled. The embedder owns the group; onejudge reports what a hook claimed on Report::processes. Install it on each provider, or on a Plan (Plan::with_spawn_hook) when you drive the CLI's run driver — the plan reaches both sides of a two-party worker + judge run. See docs/spawn-hook.md.
• Notes is the note delivery seam: a role-addressed correction sent into a running conversation reaches whichever party is live, and the other party receives it with that party's response. A note can bind a Criterion, which enters the acceptance criteria the judge evaluates against; one that arrives after the conversation completed raises Undelivered rather than being silently accepted. Open it with Notes::channel() and install the inbox on an Engine (with_notes) or a Plan. See docs/notes.md.

Two things it improves over the in-skilltest engine:

1. The judge sees tool events. Verdicts render the transcript with a compact, token-budget-aware summary of each turn's tool calls, so a criterion like "the change was committed" can be decided from the git commit the skill actually ran — not only from what it said. Transcript also exposes a ToolQuery primitive for events-backed assertions with no judge call.
2. One caller-owned session name. The engine always threads a single --session <name> across turns instead of extracting and re-passing a native id; if a harness cannot bind a session, the provider gracefully retries the call without it, re-prompting the inlined transcript.

Example

use onejudge::{Conversation, Engine, OneharnessProvider, Settings, SimulatedUser, Skill};

let provider = OneharnessProvider::new();
// Harness/model selection lives in oneharness's config files, not here; Settings
// carries only the loop's own concerns (turn cap, session name).
let settings = Settings::new();
let engine = Engine::new(&provider, settings);

let skill = Skill::new("greeter", "./skills/greeter", "Greet the user warmly.");
let user = SimulatedUser::new("A curious first-time visitor.")
    .done_when("the assistant has answered the visitor's question")
    .max_turns(6);

let outcome = engine.run(&Conversation::multi_turn(skill, "hi", user))?;

let verdict = engine.judge_boolean("the reply was welcoming", &outcome.transcript)?;
println!("{:?}: {}", verdict.value, verdict.reason);
# Ok::<(), onejudge::Error>(())

Drive a deterministic backend instead of a live harness by pointing a CommandProvider at any command that speaks the protocol.

Development

The command surface is a just recipe set; just --list is the index.

just bootstrap   # clean-clone setup: toolchain + cargo tools + fetch
just check       # the full gate: format, lint, doc, coverage-enforced tests, audit
just test        # fast unit + integration + e2e

The gate is deterministic and offline — the model is faked by real subprocess test doubles, never mocked. The one path that needs a real external service is proven in an opt-in tier, kept out of check:

• just test-live — the OneharnessProvider path against a real harness (see docs/live-tier.md).

See AGENTS.md for the durable contributor guide.

License

MIT.