reasoning-dataset-lab 0.5.0

English | 日本語 | 中文 | Español | Français | हिन्दी | Italiano | Português (BR)

A map-first lab for datasets of puzzles, quizzes and verification items, where code proves the answer key before any model words the item. rdlab is the command line. It plans a dataset as a pre-registered map, builds items whose keys are constructed and proven, asks a local model only to put them into words, and records the provenance of every item.

A key that a model chose is an opinion. A key that code built from a hidden state, and proved by a solver, is a fact about the item. rdlab is built so the key never depends on a judge.

Install

pip install reasoning-dataset-lab

Python 3.13 or newer (3.15 is the main target; CI tests 3.13, 3.14 and 3.15). It depends on aspire-si-gates and role-set.

What you can do without any model

rdlab map check examples/t10-pilot-a.map.yaml
rdlab project fingerprint . --map examples/t10-pilot-a.map.yaml
rdlab --help

• rdlab map check validates a dataset map (its axes, every cell's target against the generator's capacity, share floors and ceilings, variety bounds, acceptance criteria) and prints the map's canonical SHA-256. Exit 0 is a valid map. Exit 1 is a refusal, as a {code, message, hint} JSON object on stderr. Exit 2 is a usage or file error.
• rdlab project fingerprint hashes the five project files, the map and canon/ in a fixed order, so a build can say exactly what it was built from.
• rdlab models lists the models a Pilot B build pins, and rdlab variety computes the deterministic variety report for a build.

What needs a GPU and local models

rdlab pilot-b builds the Pilot B critic test set: triads of supported, unsupported and cannot-tell claims made from typed fact tables. It runs real local models through Ollama on loopback (a writer and two blind readers from different families), needs a GPU, and needs a licence registry file that says each model may be used for evaluation. It never pulls a model for you and never calls a cloud endpoint. The full flags, the memory plan and the num_predict settings are in the handbook.

The judge bake-off

Which local model should judge a claim? rdlab answers it the same way it builds keys: by construction, with the answers sealed until the judging is over.

rdlab bakeoff-part --output-dir <dir> ...      # build a sealed test part from certified value sets
rdlab bakeoff-probe --part-dir <dir> ...       # fail the part if its answers can be guessed from surface features
rdlab bakeoff-judge --part-dir <dir> ...       # run one judge without opening the sealed keys
rdlab bakeoff-judge-score --judge-dir <dir>    # only now join the judge's rows with the keys

Winner selection follows the pre-registration exactly, with no lenient fallbacks. scripts/colab/launch_judges.sh runs the five judges on your own Google Colab account, each in its own hash-locked environment, under a spending cap read from the live Colab rate. Splitting a counted run across several cards is built but has not completed a real run yet.

The map-first principle

1. Prove first, word second. Code constructs a formal state and proves the key before a model writes any prose. A kind of item that cannot be proved by construction is marked judged and enters only through a multi-family unanimity gate.
2. Pre-registered cells and capacity. map.yaml declares every intended cell, its target, the generator's capacity for it, share floors and ceilings, and the variety bounds, before any item exists. A cell the generator cannot yet build stays in the map with capacity: 0 and a stated reason, so a coverage gap is visible and the map is never shrunk to hide it.
3. One fingerprint. A project binds five files, the map and its canon, and a deterministic hash says which.

What has been measured

The generator's value sets (the lists a claim's values are swapped within) are certified before use. In the v3 run, 44 of 50 sets certified, against 26 before the sets were repaired. Here "certified" means two blind readers from different families called every ordered pair a contradiction, and an independent blind key had already agreed. It is evidence, not proof: model readers were part of it. The wording of a rendered item can still differ from its key, and a person's hand-read is the check for that. Nothing rdlab writes is training data until a person has checked it. The run and its files are in certification/v3.

Security and data

• Reads: the map, project files and canon you point it at.
• Writes: the output directory you give it.
• Network: the rdlab command reaches only Ollama on loopback (127.0.0.1), and only for pilot-b and models --verify-live. Anything else is refused. No telemetry.
• Opt-in Colab: scripts/colab/ uploads the items and judges you point it at to your own Colab account and downloads the results. The answer keys never leave your machine; scoring runs locally.
• Permissions: none beyond those files and that local port.

Report a vulnerability privately; see SECURITY.md.

Status

Version 0.5.0. It is alpha: interfaces and data shapes can change without notice. 1,162 tests pass (2 skipped); CI holds coverage at 90% or more.

License

MIT, © mcp-tool-shop. See LICENSE.

Built by MCP Tool Shop