conjure-eval

Public-slice harness for the CONJURE transformative-creativity benchmark. Ships the 393-instance public split (70 percent of the 560-instance Phase 4.8 frozen corpus: 510 closed-problem instances across 17 Lakatos families plus the 50-instance C4-OPEN axis of formalised open mathematical conjectures, SHA-256 a8c9842ea4d59072802689603b1e38c679fd1695194aa1cf73f81c076903daf6) so frontier-model developers can self-evaluate locally before submitting to the hidden split.

This package contains:

• The frozen public-slice corpus JSON (conjure_eval.data.public_corpus).
• A CLI for inspecting the corpus, driving a model pass, and checking submission files before they are sent to the hidden-split adjudicator.
• The deterministic split provenance, so any third party can re-derive the public/hidden split byte-for-byte from the source corpus.

What this package is and isn't

conjure-eval is a self-service developer convenience: it lets a model team inspect the public contracts, run their model against the public slice, and smoke-test their submission format before sending results to the benchmark author. It does not ship the hidden split, and it does not run the kernel-verified tight-mode adjudicator that produces the headline accept rate. Those live in the private blanc repository and are operated by the benchmark author against frozen model snapshots; the headline number reported in the brief is the hidden-split rate.

Install

pip install conjure-eval

Usage

# List all 393 public-slice instance IDs
conjure-eval list-public

# Inspect a single instance
conjure-eval show C1-bv-001

# Drive a model pass (OpenAI-compatible endpoint)
conjure-eval run \
    --base-url https://your-endpoint/v1 \
    --api-key-env MY_API_KEY \
    --model your-model-name \
    --out submissions.jsonl

# Check submission file well-formedness before sending
conjure-eval verify-submission submissions.jsonl

# Print corpus provenance fields
conjure-eval provenance

Provenance

The public corpus is a deterministic 70/30 axis-stratified slice of the 560-instance Phase 4.8 frozen corpus maintained in the private blanc repository (510 closed-problem instances + 50 C4-OPEN instances). Seed: 4317. Anyone with the source corpus can reproduce both slices via scripts/build_conjure_split.py. Every C4-OPEN instance carries a snapshot-pinned open-status certificate generated by scripts/certify_open_status.py; certificate JSONs live under instances/open_status_certificates/.

Changelog

0.3.1 (2026-05-24)

Patch: fix C4-OPEN prompt-vs-adjudicator naming mismatch.

• The v0.3.0 C4-OPEN prompts used theorem solution : ExpectedDirectProof := ..., but the adjudicator looks up declarations by exact names direct_proof, lakatos_proof, and lakatos_nonvac. Every kernel-acceptable submission therefore failed the adjudicator's name lookup. The v0.3.0 Opus 4.7 pilot registered 0/50 on both pathways purely because of this mismatch.
• v0.3.1 rewrites the C4-OPEN prompt to name declarations direct_proof, P, lakatos_proof, and lakatos_nonvac exactly, and explicitly requires the lakatos_nonvac non-vacuity obligation.
• New corpus SHA-256 a8c9842ea4d59072802689603b1e38c679fd1695194aa1cf73f81c076903daf6.
• No instance set, axis split, contract, or open-status certificate changes; patch is prompt-only.

0.3.0 (2026-05-24)

C4-OPEN axis expansion from 12 to 50 instances.

• Public corpus extends from 522 to 560 instances (393 public / 167 hidden) by adding 38 new C4-OPEN instances spanning Polignac families, prime k-tuple conjectures, Cunningham chains, Landau/Bunyakovsky polynomial families, and others.
• Every new instance ships a kernel-verified ConjectureStatement plus a snapshot-pinned open-status certificate against Mathlib commit 1ccd71f89cbb.
• Public-slice axis split: C1=134, C2=162, C3=62, C4O=35 public.
• New corpus SHA-256 c1f32624c1a698ef30d3c0a7151d69d4706c413dc910abb9f0812434cfa625c7.
• Release gate raised: check_v03_release_gate.py now requires at least 40 green certificates.

0.2.0 (2026-05-24)

CONJURE three-tier CNS plus C4-OPEN axis release.

• Public corpus extends from 510 to 522 instances (366 public / 156 hidden) by seeding a 12-instance C4-OPEN axis (Collatz, twin primes, Goldbach, Mersenne, etc.).
• Corpus schema version bumped to 2.1; new per-instance public fields: conjecture_variant and open_conjecture_statement_lean.
• Novelty test reduced from four tiers to three (SYN, LOG, CIRC); NV-VOC removed.
• New corpus SHA-256 11189adafbae46ffadabdbe4d297502472dc5ad9ee87703e1a2ca22e99229676.

0.1.0 (2026-05-23)

First public release.

• Ships the 358-instance public split (70% of the 510-instance Phase 4.6 frozen corpus, SHA-256 33e9daebbfc1382b08c4b518f6bc9b30e62c13cc9d7e178327675929ebd74cc9).
• Five CLI subcommands: list-public, show, run, verify-submission, provenance.
• No third-party dependencies; requires Python 3.10+.
• Apache-2.0 license.