orchestrator-uplift-harness 0.1.1

Harness-agnostic evaluation of single-agent and orchestrated coding systems

Orchestrator Uplift Harness

ouh compares a coding manager working alone with the same manager using model-bound subagents. Evaluation logic is independent of Codex and OpenCode; harness-specific behavior lives behind adapters.

The first comparison targets OpenCode with Gemma 4 31B as the manager and mistralai/devstral-2512 as the optional coding worker. The checked-in condition keeps the canonical model ID openrouter/mistralai/devstral-2512. Because current OpenCode model discovery does not expose that entry, the adapter creates an isolated OpenRouter-compatible provider alias for transport and records both identities in every trial.

Quick start

Install the published CLI and copy its immutable input suite into a writable working directory:

uv tool install orchestrator-uplift-harness
ouh init --destination ./orchestrator-eval
cd orchestrator-eval
ouh validate --root .

For development from a source checkout:

uv sync --python 3.14
uv run ouh validate --root .
uv run ouh preflight \
  --comparison comparisons/opencode_gemma4_devstral2.toml
uv run ouh run \
  --comparison comparisons/opencode_gemma4_devstral2.toml \
  --suite suites/spawn_smoke.toml \
  --repeats 1 \
  --run-id opencode-spawn-proof
uv run ouh report --run-id opencode-spawn-proof

Coding-Harness auswählen

Codex und OpenCode sind als Adapter verfügbar, aber sie laufen nicht gleichzeitig. Der Pfad hinter --comparison wählt genau einen Harness. Die Comparison verweist auf zwei Condition-Dateien – Single-Agent-Baseline und Orchestrator-Kandidat – und beide müssen denselben harness-Wert besitzen.

OpenCode mit Gemma 4 allein gegen Gemma 4 plus Devstral:

ouh preflight --comparison comparisons/opencode_gemma4_devstral2.toml
ouh run \
  --comparison comparisons/opencode_gemma4_devstral2.toml \
  --suite suites/smoke.toml \
  --repeats 3

Codex Sol allein gegen Sol mit den konfigurierten Workern:

ouh preflight --comparison comparisons/codex_sol_workers.toml
ouh run \
  --comparison comparisons/codex_sol_workers.toml \
  --suite suites/smoke.toml \
  --repeats 3

Die indirekte Auswahlkette lautet:

--comparison
    -> baseline/candidate Condition-Dateien
    -> harness = "opencode" oder harness = "codex"
    -> ausschließlich dieser Adapter führt beide Trial-Seiten aus

Die CLI nennt den aktiven Harness in der Preflight- und Run-Ausgabe sowie in run.json. Ein Vergleich, der Codex und OpenCode mischt, wird bei der Konfigurationsvalidierung abgelehnt. Alternative Binary-Pfade können über OPENCODE_BIN, CODEX_BIN und CODEX_HOME gesetzt werden.

Use suites/smoke.toml and multiple repeats for an uplift experiment. The one-case spawn_smoke.toml suite proves the wiring only; it cannot establish a quality advantage.

The native smoke suite now spans eight task shapes, including concurrency, graph/cycle debugging, backward-compatible feature work, and security hardening. For real-repository evidence, benchmarks/ contains pinned, deterministically stratified SWE-bench Verified and Multilingual selections. The package ships selection metadata and digests, not upstream repositories, test patches, gold patches, model outputs, or run artifacts.

Included inputs

• Eight deterministic native coding tasks and hidden workspace graders.
• Codex and OpenCode single-agent/orchestrated conditions.
• OpenCode Gemma 4 31B plus mistralai/devstral-2512 worker condition.
• SWE-bench Verified selections with 24 and 100 pinned instance IDs.
• SWE-bench Multilingual selections with 18 and 42 pinned instance IDs.
• The checkout and official Docker-evaluator bridge for real SWE-bench issues.

All benchmark families remain separate in reports. Function synthesis, repository issue resolution, and terminal work are not collapsed into one opaque score.

Evaluation contract

• Baseline and candidate receive the same task and fresh fixture state.
• The baseline technically denies child creation.
• Delegation-required cases must produce a child session; negative controls must not delegate.
• Quality comes only from an external deterministic workspace grader.
• Runtime status, quality, tokens, cost, latency, and cleanup are stored as separate observations.
• Every observed child records its agent, canonical model, transport model, parent session, and exact ID.
• OpenCode state is isolated per trial. Cleanup deletes only observed session IDs and verifies that the trial database contains none of them afterward.
• Large or truncated OpenCode exports fall back to the same isolated session database for identity and usage; failure of both sources blocks the trial.

Generated runs live under artifacts/<run-id>/ and are intentionally ignored by Git because they may contain large transient workspaces. The final report is available as JSON, Markdown, and static HTML inside that directory.

Requirements

• Python 3.14 or newer.
• Git for isolated benchmark workspaces.
• OpenCode and/or the Codex app binary for their respective adapters.
• Docker plus a separately installed, pinned SWE-bench evaluator only when running external SWE-bench instances.

Provider credentials remain in the native harness credential stores. They are not copied into the generated working directory or distribution archives.

License

MIT. Upstream benchmark repositories and datasets retain their own licenses.

Development

uv sync --python 3.14
uv run pytest
uv run ruff check .
uv run ruff format --check .
uv run mypy src

See the versioned design and implementation plan under docs/superpowers/.