cordis-bench

cordis-bench benchmarks whether language models understand spatiotemporal lifecycle/effect composition well enough to reason about application-visible consequences and use a Cordis harness correctly.

Current paper target: V2.0.1

V2.0.1 uses formal + actual Cordis crossed with core + challenge. Within a realization, core and challenge use the same primary task contracts and answer formats; semantic size is the intended axis that changes.

V2.0.1 identifier surface

V2.0 used long randomized opaque identifiers such as plugin_849065282 and slot_689631612. V2.0.1 keeps the identifiers semantically neutral but renders them as short opaque symbols such as plugin_a, provider_k, service_b, slot_q, and formal capability keys such as cap_c.

The naming patch consumes exactly the same RNG draws as V2.0. For a fixed seed, the semantic construction is therefore preserved: effect programs, numeric values, interference structure, schedules, targets, task allocation and size ladder are unchanged apart from alpha-renaming. Because prompts and gold identifier strings change, V2.0 model predictions are not reused as V2.0.1 scores.

Paper release size

The paper-facing V2.0.1 release is 1,200 exact-output questions assembled from three independent 400-item generation blocks with seeds 0, 1, 2.

The release contains 1,056 primary questions and 144 challenge-only outcome-count diagnostics, generated from 240 independent latent systems. The semantic-size ladder remains 2,4 | 8,16,24,32; scaling is replication, not a new difficulty axis.

Generate V2.0.1

python -m pip install -e '.[dev]'
npm install --prefix native/cordis --ignore-scripts --no-audit --no-fund

# Paper release: seeds 0, 1, 2 -> 1,200 rows.
cordis-bench generate --version 2 --seed 0 \
  --output data/generated/v2.0.1-release.jsonl

Run the shortcut audit before freeze:

python -m cordis_bench.v2_audit data/generated/v2.0.1-release.jsonl \
  --output results/v2.0.1-release-shortcut-audit.json

Probe and score

Use an explicit 8,192-token completion cap for every V2.0.1 run:

cordis-bench probe data/generated/v2.0.1-release.jsonl \
  --model google/gemini-3.7-flash \
  --max-tokens 8192 \
  --output results/gemini-v2.0.1-release.jsonl

cordis-bench score data/generated/v2.0.1-release.jsonl \
  results/gemini-v2.0.1-release.jsonl \
  --output results/gemini-v2.0.1-release-score.json

cordis-bench probe data/generated/v2.0.1-release.jsonl \
  --model openai/gpt-5.6-luna --effort low --max-tokens 8192 \
  --output results/luna-v2.0.1-release.jsonl
cordis-bench score data/generated/v2.0.1-release.jsonl \
  results/luna-v2.0.1-release.jsonl \
  --output results/luna-v2.0.1-release-score.json

# Explicit Albert routing is free for authorized Inria accounts and never
# falls back to the paid OpenRouter route. Albert allows 50 RPM; use 35 for margin.
cordis-bench probe data/generated/v2.0.1-release.jsonl \
  --model albert/deepseek-v4-flash --effort low --max-tokens 8192 \
  --workers 8 --rpm 35 --timeout 600 \
  --output results/deepseek-v4-flash-albert-v2.0.1-release.jsonl
cordis-bench score data/generated/v2.0.1-release.jsonl \
  results/deepseek-v4-flash-albert-v2.0.1-release.jsonl \
  --output results/deepseek-v4-flash-albert-v2.0.1-release-score.json

Model calls go through litlm; --workers bounds concurrency and --rpm optionally limits request starts. Each settled success is flushed immediately to the output JSONL, so rerunning the same command resumes unfinished rows. Provider-prefixed routes are resolved by litlm: albert/... uses ALBERT_API_KEY, while openrouter/... uses OPENROUTER_API_KEY. There is no cross-provider fallback.

V2 uses structured outputs (integer, scalar_sequence, string_set); there is no LLM judge and no candidate-option surface. The primary_metrics report uses Jaccard for set tasks, per-observable accuracy for schedule prediction and executed success for reconfiguration. Exact match and parse rate remain strict diagnostics.

Actual Cordis validation

Native tasks execute against pinned cordis@4.0.0-rc.7, source commit 56b3d4f725681cf4556c1a8695a709cc3b6eed74. Every retained native record must agree with the finite reference. Reconfiguration answers are executed at scoring time and must reach the requested target, preserve unrelated state and use the minimum action count. Independent native cases are sharded across four isolated Node processes; each process executes its cases serially so Cordis runtime state is never shared across concurrent cases.

The benchmark is not testing whether Cordis itself is unsafe. Controlled completion orders expose application-visible consequences of lifecycle/effect composition so model reasoning can be evaluated.

V2.0 historical results

The checked-in data/generated/v2-release.jsonl and matching Gemini/Luna release results belong to V2.0, with the long numeric identifier surface. They remain historical evidence and a useful paired comparison, but they must not be mixed with V2.0.1 rows or scores. V2.0.1 requires fresh evaluation because the prompts changed.

See V2.md, EVALUATION_V2.md, and V1_TO_V2_PAPER_CHANGELOG.md.

Historical V1.x

V1.x remains frozen for reproducibility. The old quantitative MCQ helper derived distractors from the gold, compromising the nominal 25% answer-surface control for affected questions. This does not invalidate the underlying semantic worlds or exact oracles. V2 removes candidate answer sets rather than patching distractor heuristics.