Orchestrator Evaluation Harness

Installable, contract-driven comparison of two software-development orchestrator models. It supports OpenRouter and other OpenAI-compatible Chat Completions endpoints, including different endpoints for the two candidates and the coding worker.

The package contains the v2 contract gateway, deterministic Oracle arm, live worker arm with replay cassettes, six validated task/scenario pairs, budget ledger, paired statistics, Markdown/JSON reporting, and neutral evidence-bundle export. It contains no historical results.

Install

Build locally:

python3 -m build
python3 -m pip install dist/orchestrator_eval_harness-0.2.0-py3-none-any.whl

Or install the source checkout:

python3 -m pip install .

Create an editable configuration:

orchestrator-eval --output experiment.json init-config

Configure endpoints and models

Credentials are never stored in JSON. api_key_env names the environment variable that contains the bearer token. Set it to null for an endpoint that does not require authentication.

{
  "experiment_id": "candidate-a-vs-candidate-b",
  "endpoints": {
    "openrouter": {
      "url": "https://openrouter.ai/api/v1/chat/completions",
      "api_key_env": "OPENROUTER_API_KEY",
      "provider_routing": true,
      "include_usage": true,
      "headers": {
        "HTTP-Referer": "https://localhost/orchestrator-eval",
        "X-Title": "Orchestrator Evaluation Harness"
      }
    },
    "local": {
      "url": "http://127.0.0.1:1234/v1/chat/completions",
      "api_key_env": null,
      "provider_routing": false,
      "include_usage": false,
      "headers": {}
    }
  },
  "orchestrators": {
    "candidate_a": {
      "endpoint": "openrouter",
      "model": "vendor/model-a",
      "input_price_per_token": 0.0000001,
      "output_price_per_token": 0.0000004,
      "reasoning": {"enabled": true}
    },
    "candidate_b": {
      "endpoint": "local",
      "model": "model-b",
      "input_price_per_token": 0,
      "output_price_per_token": 0,
      "reasoning": {}
    }
  },
  "worker": {
    "endpoint": "local",
    "model": "coding-worker",
    "input_price_per_token": 0,
    "output_price_per_token": 0,
    "reasoning": {}
  }
}

Keep the remaining limits and statistical parameters from the generated template. Exactly two orchestrators are required. Static authorization, cookie, and API-key headers are rejected; secrets must come from environment variables.

provider_routing: true enables OpenRouter provider discovery and pins each model to its discovered provider for scored calls. Generic endpoints should normally use false. include_usage: true sends OpenRouter's explicit usage request field; leave it false for generic OpenAI-compatible servers.

Run

Offline checks require no credentials:

orchestrator-eval --config experiment.json validate
orchestrator-eval --config experiment.json power
orchestrator-eval --config experiment.json --results-dir /tmp/eval-dry dry-run

Live evaluation:

export OPENROUTER_API_KEY="..."
orchestrator-eval --config experiment.json run

By default, results are written to ./results/<experiment_id>/. Use --results-dir to choose another location. Stages are resumable:

orchestrator-eval --config experiment.json smoke
orchestrator-eval --config experiment.json screen
orchestrator-eval --config experiment.json e2e
orchestrator-eval --config experiment.json report

Custom task/scenario sets can be supplied together:

orchestrator-eval \
  --config experiment.json \
  --tasks-dir ./my-tasks \
  --scenarios-dir ./my-scenarios \
  validate

Both directories must contain matching unique id and class fields.

Analyze an existing run

report recomputes the paired statistics and writes report.json plus report.md. It does not call a model:

orchestrator-eval \
  --config experiment.json \
  --results-dir ./results/candidate-a-vs-candidate-b \
  report

Export the raw run rows, screening rows, configuration, report, ledger, provider pins, cassette statistics, and integrity hashes into one neutral JSON file:

orchestrator-eval \
  --config experiment.json \
  --results-dir ./results/candidate-a-vs-candidate-b \
  bundle

What the evaluation measures

The primary orchestration arm uses a deterministic Oracle worker, isolating delegation, ownership, verification, correction, and finalization behavior. The secondary system arm uses the configured coding worker behind a cassette store. Provider failures are recorded as infra_error and excluded from model success rates.

Every manager tool call is locally schema-validated. Delegated paths must be authorized and disjoint, acceptance checks must name manifest checks, worker test claims are checked, and finalization requires fresh harness-issued verification evidence.

The decision rule requires the configured minimum number of complete pairs, a paired bootstrap interval excluding zero, the configured practical effect size, and exact McNemar significance.

Verify a distribution

python3 scripts/verify_distribution.py dist/*.whl dist/*.tar.gz

The verifier rejects archives containing result directories, workspaces, cassettes, generated reports/bundles, caches, or secret-looking API keys.