gsj-harness-rollout-server

A rollout server for the gsj corpus: given a task (case, timestep, prompt), it runs a pinned coding agent (pi 0.83.0) in an isolated sandbox whose git checkout and retrieval are both truncated at timestep, captures every token and logprob the model produced, and emits one validated, training-ready trajectory — task → sandbox → agent → trace, nothing else (no storage, scheduling, rewards, weights, versioning, or training; those belong to the trainer that calls it).

What these traces are for, today. Training-ready names the trace's shape — token ids, a loss mask and the captured logprobs, the form a trainer consumes. What has been proven is narrower, and each part of it is measured below or in the verdict and charter it links:

• Sound now, for provenance work — every claim the validators make. The cutoff holds: a forged timestep claim is refused 401 from inside the sandbox, and gate G5 audits every retrieved page ≤ T from the trace alone. The pins are checked — the system prompt, the skill cards, the tool roster, the thinking-mode tail and the no-compaction settings, each against its pinned value, by the same checks.py on both sides of the wire. The reconstruction is faithful: loss_mask exact at zero tolerance and prompt_ids byte-identical against the predecessor's golden reference, on the Mac pair and again on the H200 pair, and the logprobs are real captures.
• Not in the trace — the estate's to record, the consumer's to account for. The sampling policy: pi sends no sampling parameters, so the engine's configuration is the policy; no OpenAI-compatible API reports it, and on a borrowed endpoint the temperature that produced the logprobs is recorded nowhere (charter §7 row 22 — open, half by decision, half owned by the first production bring-up). The codec too: no trace carries it — the pins walk verified it at bring-up on the reference estate, and a borrowed endpoint exposes neither the tokenizer's nor the template's bytes. The reward is the trainer's outright, like storage and weight sync: every callback carries reward: null, and the citation reward was deferred at CP-88 rather than stacked on the page grader, which pays for form — 1.0 for a bare list of rubric headings with in-cutoff page numbers, checking nothing about whether those pages were read.
• Trained: single optimizer steps and two two-step runs — no training result. Single steps in the slime (CP-17) and verl (CP-21) loops, a stranger's loop (CP-26), the first thinking-on run (CP-32, 72 episodes) and one full-archive step (CP-87); and two two-step runs on the H200 — CP-69 (reward 1/191 → 78/128, which the record reads as near-verbatim format-copying of a seed trajectory: CP-17's mode-collapse onset, measured at scale) and CP-87 (192 episodes per collection, 1/192 → 75/192: a templating policy adding page numbers to the heading list the grader pays for, not a substantive improvement). The verdict refuses to call the first loop a training result, and the record reads none of them as substantive learning.

Episode execution and trajectory reconstruction are NVIDIA's Polar, vendored by commit with three carried patches; our own code is the 2,034-line shell that points Polar at our corpus, retrieval service, agent, and checks. The evaluation behind it ended in ADOPT (provisional 2026-08-09, converted 2026-08-11 on production hardware) — docs/VERDICT.md is the standalone statement, with the conditions that would reverse it.

_{One task triple in, one validated trace out. Everything around the sandbox is Polar's, the operator's estate, or the trainer's; ours is the harness, the builder, the receiver, and the checks.}

Install — three routes, by what each one gets you

The wheel is not the server. It carries the trainer client, the validators, the corpus pipeline and the estate tool; episode execution is NVIDIA's Polar, which publishes no Python artifact of any kind, so only the third route below runs an episode.

1 · A trainer client — runs no episodes. It submits tasks to a server somebody else operates, polls them, and re-checks every trace that comes back with the same checks the server ran (RolloutClient, checks, gsj-rollout submit). It cannot start a sandbox or stand an estate. Any Python ≥ 3.12; three dependencies, pydantic, httpx, pyyaml.

python3 -m venv .venv && . .venv/bin/activate   # stock Ubuntu 24.04 lacks python3-venv (the venv fails: ensurepip is not available): `sudo apt install python3-venv` first
pip install gsj-harness-rollout-server
# not a bare `pip install` outside a venv: Ubuntu >= 23.04 refuses it — `error: externally-managed-environment` (PEP 668) — and stock 24.04 has no pip outside one at all

2 · An estate, no Polar — stands the services up, and runs no episode by itself. From a corpus directory, one command validates the corpus and stands up Forgejo (one repository per case, one branch per timestep), the MCP retrieval service with the corpus ingested, the taskbank, a validated rollout.yaml and a pins skeleton; the receiver (gsj-rollout serve) runs from the wheel too. What it cannot do is execute an episode — up's closing block says so and names both routes to Polar. Needs Docker with the Compose v2 plugin, git, and your model endpoint.

docker pull ghcr.io/mhganainy/gsj-pi-harness:pi0.83.0-3   # the sandbox image: `up` checks for it and never pulls it
pip install gsj-harness-rollout-server pyarrow           # in the venv above; pyarrow is the taskbank's parquet writer, deliberately not a core dependency (ADR-0022 §5) — `up` refuses without it
mkdir -p runs                                            # `up` never creates its runs root (CP-90); from a wheel it is ./runs
python -m gsj_rollout.estate up --corpus <root> --engine-url <endpoint> --engine-model <served model> -y

3 · A server that runs episodes — route 2 plus Polar's two processes. The wheel carries neither way to get Polar:

• the published image ghcr.io/mhganainy/gsj-polar:f0e8343a-gsj0.1.17 — public, anonymously pullable, cut at every release from this repo's vendored Polar plus that release's PyPI wheel (A-28). The two docker run invocations that start its processes, one clause per flag, are on bring your own § Polar's two processes; no clone.
• a checkout, with Polar's own venv — which must also host gsj_rollout, because Polar loads our harness and builder by import path:

# prerequisite: `uv` — a fresh Ubuntu has none (exit 127 at `uv venv`): `pip install uv` in the venv, or `curl -LsSf https://astral.sh/uv/install.sh | sh`
git clone https://github.com/MHGanainy/gsj-harness-rollout-server && cd gsj-harness-rollout-server
python3.12 -m venv .venv && . .venv/bin/activate && pip install -e ".[dev]" uv
cd vendor/polar && uv venv --python 3.12 .venv && uv pip install -p .venv/bin/python -e . && uv pip install -p .venv/bin/python -e ../.. && cd ../..
gsj-rollout serve --config rollout.yaml    # renders topology.rendered.yaml, prints the two Polar commands, runs the receiver

Quick start

from gsj_rollout import RolloutClient, load_config
from gsj_rollout.config import render_task_request
cfg = load_config("rollout.yaml")            # one YAML describes the whole deployment
request = render_task_request(cfg, task_id="demo", instruction="Fix the failing test.", case_id="case_0001", timestep=12)
traces = RolloutClient(cfg.polar.rollout.base_url).collect([request])   # Polar's rollout API, not the receiver; list[Trace], only checks-clean sessions come back

gsj-rollout submit --config rollout.yaml --case case_0001 --timestep 12 --prompt "Fix the failing test."
# exit codes: 0 all collected · 1 partial · 2 usage · 3 server unreachable

The two roles

One trap worth naming at the door. checks validates traces against pinned approved sets (tool rosters, system prompts, skill cards, settings), and the wheel ships this estate's pins — on any other estate every hash gate fails *_not_approved, loudly, by design. Point GSJ_PINS_PATH at your own pins file before the first import of gsj_rollout.checks; resolution is GSJ_PINS_PATH → repo checkout → packaged copy (a UserWarning when the packaged copy is what resolved), and an unusable path raises PinsConfigurationError rather than falling through. The thinking-on reference set rides at gsj_rollout/pins/thinking-on/pins.gsj.json — gate G6 compares against per-mode pins data. Format and reasoning: docs/checks-spec.md.

The timestep cutoff

_{timestep is a boundary the agent cannot cross — one wall on the filesystem, one in front of retrieval — enforced twice and audited once.}

• The filesystem wall. The sandbox clone is branch timestep-T, --depth 1, remote removed, reflogs scrubbed — git history cannot reach a page past T even offline.
• The retrieval wall. The harness mints an HS256 token host-side with claims {case_id, timestep, episode_id, exp}; the signing secret never enters the sandbox; the service verifies the signature, then filters to page ≤ T before ranking, with T from the verified claims only. The agent may read its own token but cannot widen its timestep — any mutation invalidates the signature (tampered claims, timestep 12→18 with the original signature: HTTP 401).
• The audit. checks.py gate G5, from the trace alone: every retrieved page ≤ T; the checkout shallow, zero remotes, branch == timestep-T; checkout pages contiguous 1..T.

Documentation

• The guide, seven flat pages under docs/guide/: index · how it works · validation and pins · server guide · trainer guide · bring your own model and corpus · troubleshooting
• Normative: Verdict (read first) · Charter (scope laws; assumptions §4; gap register §7; standing rules §8) · Checks spec · Corpus contract

What has been proven

The badge covers none of this — not the golden pairs, fidelity, the loops, or any episode (episodes need an estate and GPU time). Green means the fixtures still pass, not that the harness runs.

What it does not do

• It has taken optimizer steps and trained nothing, and says so. Each loop in the table above (CP-17, CP-21) is exactly one optimizer step bracketed by two collections, and the post-sync 8/8 reward reads as the onset of mode collapse, not competence; the two two-step runs since (CP-69, CP-87) read the same way at scale — reward uptake on form, not substance. Concurrent collection-and-training and weight sync at cadence have zero data points.
• The trainer's problems stay the trainer's: storage, retention, mixing, staleness, collation, reward — every callback carries reward: null — and weight sync. Dropped deliberately, at the start (charter §7 rows 16–21).
• Sampling and codec provenance are the estate's, not the trace's. pi sends no sampling parameters, so the engine's configuration is the sampling policy — an unpinned engine silently samples at neutral defaults (measured). Codec identity is verified at bring-up by the pins walk, not per-trace.
• Two open gaps, of a 32-row capability register (21 parity, 7 dropped by decision, 1 better, 1 TBD): row 12 — G4 codec evidence never rides the callback, receiver-side by decision; row 22 — per-episode binding of traces to engine identity (serve argv, generation config, codec), owned by the first production bring-up. The separate anonymous-read gap is closed: the reference estate and demo require Forgejo sign-in and use credentialed clones (CP-58/59; re-measured at CP-82). This repository defense leaves the engine and MCP reachable, as episodes require; it does not close row 22's provenance gap.
• Model-agnostic in mechanism, Qwen-fitted in defaults. One foreign family is one data point; a different reasoning geometry is the wall. Thinking-on requires the symmetric served template.

Repository layout

pytest -q requires the Docker CLI and Compose plugin for credential parsing checks; no Docker daemon or running estate is needed.

The CP-06 feasibility spike (stub backend, spike harness, the stub-side wire captures, p1_verdict.py, wire_diff.py) left the tree at CP-68 and is frozen at tag spike-cp06 — commit a769771, tree 2082a24f; restore with git checkout spike-cp06 -- spike.

Where the record lives

The development record is maintained privately. The per-checkpoint reports and prompts, the 43 ADRs, the 2026-08-24 adversarial audit (24 agents; no live gate, pin value, or code path wrong; zero of 91 findings refuted), and the raw docs/golden//docs/polar/ artifacts exist unchanged in the operator's working tree, not in this repository — so the evidence column above, the shelf map in docs/README.md, and two report paths in the shipped pins.gsj.json provenance resolve only there; for anyone else those claims reduce from "checkable at the cited path" to "asserted". What stays checkable from a clone: the code, the three suites, the pins walk, the wheel assertions, and the evidence bodies they execute against.

Licence

Apache-2.0 — LICENSE. vendor/polar/ is NVIDIA's, carries its own Apache-2.0 LICENSE, and ships in no released artifact of ours — no wheel, no sdist; it does reach you inside the demo's gsj-polar image, which is built from it (CP-99: a stranger quoted this sentence back as "Polar ships in no released artifact" while a public image carrying it was one docker pull away). The wheel contains gsj_rollout/, the two pins sets, the G2 reference capture (pins/container/system_prompt.container.derived.txt), ingest_corpus.py and estate.py (as gsj_rollout.ingest_corpus / gsj_rollout.estate — gsj_rollout.bringup on wheels 0.1.3–0.1.5) — nothing else, asserted at build time (18 entries since 0.1.3). Predecessor: gsj-envloader @ v0.8.0, archived 2026-08-25 — still the goldens' collecting stack, readable at v0.8.0; no longer the fallback, a term that expired at the verdict's conversion on 2026-08-11. Its frozen sandbox/ recipe also supplied the harness image's arm64 build at CP-64, and its staging/BRINGUP.md remains the cold-start reference linked from the living estate recipe. Reuse is recorded here; the archive stays frozen.

Provenance

• The front door (CP-105): what the traces are for, moved above the fold from the sections below it, the verdict (the CP-17 loop "not a training result") and the charter (§7 row 22; the single steps at CP-26, CP-32 and CP-87, the CP-69 and CP-87 two-step runs, and the CP-88 phase-6 deferral) — no claim in it new; the install routes labelled by what each gets you, the wheel-runs-no-episode statement having reached the door at CP-99; the one new fact, measured at CP-105 on ubuntu:24.04: stock 24.04 lacks the venv package, so the venv line fails until python3-venv is installed.
• The verdict provisional at CP-12, converted at CP-17 · the predecessor archived at CP-45 (2026-08-25, ADR-0026), the fallback term expired at CP-17 (2026-08-11) · the wheel's contents asserted at build time — CP-19, ingest_corpus.py in the wheel since CP-34.
• The filesystem wall — CP-11 · gate G6 as per-mode pins data, the thinking-on set in the wheel — ADR-0024 · the two ARM image pulls registered — wishlist row 40 · an unpinned engine at neutral defaults — CP-09 finding F1 · codec at bring-up via the pins walk, G4 estate-side — ADR-0011.
• One optimizer step per loop, 8/8 read as mode-collapse onset — CP-21 · zero data points on concurrent collect-and-train and cadence sync — charter A-13 · the trainer's problems dropped — CP-00, charter §7 rows 16–21 · row 22 and credentialed-clone/egress owned by the first production bring-up — CP-40 · "Qwen-fitted in defaults" — CP-38's own words · the symmetric served template — charter A-22.