whetstone-envs

Reproducible quick-test environment contracts and task families.

Scope

This repo owns the environment data and evaluation rules shared by Whetstone's quick-test task families, with no dependency on optimizer or execution-contract code:

• Instances define immutable task inputs, private gold data, generation seeds, task strata, and public prompt identity.
• Pools and splits validate ordered instance collections and allocate deterministic internal, official, and held-out cohorts.
• Probes pair naive and ceiling templates, render public prompt inputs, and normalize predictions for evaluation.
• Scoring represents scored, failed, and missing observations and aggregates complete repeat matrices through task, stratum, and overall levels.
• Manifests pin generated pools with versioned identities and bounded canonical persistence.
• C11 JSON canonicalization provides deterministic RFC 8785 tasks, an independent canonicalization oracle, and naive and known-good probes.
• C18 PrOntoQA provides deterministic fictional-ontology entailment tasks with an independent forward-chaining oracle.
• C19 MiniGrid state prediction provides deterministic grid-world tasks, a supported answer-relevant physical-state transition oracle, and naive and known-good probes.
• C22 instruction constraints provides fixed seeded pools of composed IFEval constraints and strict all-pass scoring.
• C23 subregular induction provides determinate hidden-rule string transformations across four ISL and OSL strata.

Task-family implementations live in their owning subpackages alongside the shared harness; the adapter to Whetstone's optimizer lives above this package.

Installation

uv add whetstone-envs

Install C18's pinned generator dependencies when generating its pools:

uv add 'whetstone-envs[c18]'

Instances

whetstone_envs.instances owns the immutable unit passed through generation, prompting, scoring, splitting, and persistence. Prompt inputs are public; gold remains private evaluation data.

@dataclass(frozen=True, slots=True)
class Instance:
    id: str
    seed: int
    strata: tuple[str, ...]
    prompt_inputs: Mapping[str, str] = field(default_factory=lambda: ...)
    gold: str = ""

def make_instance(
    *,
    id: str,
    seed: int,
    strata: tuple[str, ...] | str,
    prompt_inputs: Mapping[str, str] | None = None,
    gold: str = "",
) -> Instance: ...

def public_prompt_identity(
    instance: Instance,
) -> tuple[tuple[str, str], ...]: ...

Pools and splits

whetstone_envs.pools owns validated ordered pools and the deterministic policy for selecting three disjoint evaluation cohorts. Split optimization is delegated to dr-graph; returned instances preserve pool order.

@dataclass(frozen=True, slots=True)
class PoolSplit:
    internal_eval: tuple[Instance, ...]
    official: tuple[Instance, ...]
    held_out: tuple[Instance, ...]

@dataclass(frozen=True, slots=True)
class TaskPool:
    instances: tuple[Instance, ...]

    @property
    def strata(self) -> tuple[str, ...]: ...

    def stratum_counts(self) -> dict[str, int]: ...
    def in_stratum(self, label: str) -> tuple[Instance, ...]: ...
    def split(
        self,
        internal_eval_n: int,
        official_n: int,
        held_out_n: int,
    ) -> PoolSplit: ...

Probes

whetstone_envs.probes owns the floor/ceiling prompt pair and the default renderer that can see only public prompt inputs. Normalization strips whitespace and complete outer triple-backtick fences.

def render_with_prompt_inputs(template: str, instance: Instance) -> str: ...
def normalize(prediction: str) -> str: ...

@dataclass(frozen=True, slots=True)
class ProbePair:
    naive_template: str
    ceiling_template: str
    render: Callable[[str, Instance], str] = render_with_prompt_inputs

    def render_naive(self, instance: Instance) -> str: ...
    def render_ceiling(self, instance: Instance) -> str: ...

Scoring

whetstone_envs.scoring keeps failures and absent results distinct from binary scores. Aggregation exposes a mean only when the complete planned task/repeat matrix is present and scored.

@verify(UNIQUE)
class Outcome(StrEnum):
    SCORED = "scored"
    FAILED = "failed"
    MISSING = "missing"

@dataclass(frozen=True, slots=True)
class Observation:
    task_id: str
    repeat_id: int
    outcome: Outcome = Outcome.SCORED
    score: int | None = None

@dataclass(frozen=True, slots=True)
class Aggregate:
    mean: float | None
    usable: int
    failed_count: int
    missing_count: int
    label: str | None = None
    children: tuple["Aggregate", ...] = field(default_factory=tuple)

def aggregate(
    observations: Iterable[Observation],
    task_strata: Mapping[str, tuple[str, ...]],
    *,
    expected_repeat_ids: Iterable[int],
) -> Aggregate: ...

exact_match, scored, failed, and missing provide the primary leaf-level constructors. aggregate_task, aggregate_stratum, and aggregate_overall expose the individual aggregation steps when callers already own the hierarchy.

Manifests

whetstone_envs.manifests owns the serialized boundary for regenerated pool identity. Manifests use a closed Pydantic schema, dr-serialize identities, and dr-store canonical files.

class Manifest(BaseModel):
    generator_version: str
    seed_range: tuple[int, int]
    stratum_counts: Mapping[str, int]
    content_hash: Sha256Digest
    schema_version: int = MANIFEST_SCHEMA_VERSION

    @classmethod
    def from_pool(
        cls,
        pool: TaskPool,
        *,
        generator_version: str,
        seed_range: tuple[int, int],
    ) -> "Manifest": ...

    def write(self, path: Path) -> None: ...
    @classmethod
    def read(cls, path: Path) -> "Manifest": ...
    def matches_pool(self, pool: TaskPool) -> bool: ...

def content_hash(pool: TaskPool) -> Sha256Digest: ...

C11 JSON canonicalization

whetstone_envs.c11 generates balanced adversarial tasks for RFC 8785 whitespace removal, key ordering, number rendering, Unicode escaping, and mixed inputs. An independent, exactly pinned oracle produces private gold; the shared harness owns splitting, prompting, scoring, and persistence.

@verify(UNIQUE)
class C11Stratum(StrEnum):
    WHITESPACE = "c11/whitespace"
    KEY_ORDER = "c11/key-order"
    NUMBER = "c11/number"
    UNICODE = "c11/unicode"
    MIXED = "c11/mixed"

DEFAULT_SPLIT_SIZES: tuple[int, int, int]
PROBES: ProbePair

def generate_pool(
    *,
    n_per_stratum: int = ...,
    seed_start: int = ...,
) -> TaskPool: ...

def build_manifest(pool: TaskPool) -> Manifest: ...
def canonicalize(input_json: str) -> str: ...

C18 PrOntoQA

whetstone_envs.c18 provides deterministic fictional-ontology deductive-entailment pools. An independent forward-chaining oracle derives each label from public question and query text before an instance enters the pool.

@verify(UNIQUE)
class DistractorMode(StrEnum):
    NONE = "none"
    RELEVANT = "relevant"

@dataclass(frozen=True, slots=True)
class DepthStratum:
    hops: int
    distractors: DistractorMode

@dataclass(frozen=True, slots=True)
class SplitPlan:
    internal_eval: int
    official: int
    held_out: int

@dataclass(frozen=True, slots=True)
class GenerationConfig:
    generator_version: str
    seed_start: int
    n_per_stratum: int
    strata: tuple[DepthStratum, ...]
    split: SplitPlan

DEFAULT_CONFIG: GenerationConfig
HARD_CONFIG: GenerationConfig
PROBES: ProbePair

def generate_pool(
    config: GenerationConfig = DEFAULT_CONFIG,
    *,
    n_per_stratum: int | None = None,
) -> TaskPool: ...

def default_split_sizes(
    pool: TaskPool,
    config: GenerationConfig = DEFAULT_CONFIG,
) -> tuple[int, int, int]: ...

def build_manifest(
    pool: TaskPool,
    config: GenerationConfig = DEFAULT_CONFIG,
) -> Manifest: ...

def score_gold(prediction: str, gold: str) -> int: ...

The frozen default and hard configurations use a pinned vendored PrOntoQA generator. Their checked-in manifests pin the complete pool content; custom validated configurations produce explicit, unpinned cohorts. Regeneration is a repository operation:

uv run python scripts/regenerate-c18.py \
  --config default \
  --output src/whetstone_envs/c18/resources/default.manifest.json

C19 MiniGrid state prediction

whetstone_envs.c19 generates balanced navigation, object- manipulation, and door-interaction tasks on 5x5 and 8x8 MiniGrid worlds. Its independent oracle simulates complete LRFPDT scripts from the public grid and is checked against the pinned MiniGrid adapter after every action prefix.

@verify(UNIQUE)
class Action(StrEnum):
    LEFT = "L"
    RIGHT = "R"
    FORWARD = "F"
    PICKUP = "P"
    DROP = "D"
    TOGGLE = "T"

@verify(UNIQUE)
class C19Fact(StrEnum):
    COORDINATE = "coordinate"
    HEADING = "heading"
    FRONT = "front"
    CARRYING = "carrying"

@verify(UNIQUE)
class C19Scenario(StrEnum):
    NAVIGATION = "navigation"
    MANIPULATION = "manipulation"
    DOOR = "door"

@verify(UNIQUE)
class C19Size(IntEnum):
    SMALL = 5
    MEDIUM = 8

DEFAULT_SPLIT_SIZES: tuple[int, int, int]
PROBES: ProbePair

def generate_pool(
    *,
    n_per_stratum: int = ...,
    seed_start: int = ...,
) -> TaskPool: ...

def build_manifest(
    *,
    n_per_stratum: int = ...,
    seed_start: int = ...,
) -> Manifest: ...

def derive_fact(grid_text: str, command: str, fact: C19Fact) -> str: ...

C22 instruction constraints

whetstone_envs.c22 provides two fixed, seeded pools of composed Google Research IFEval constraints. The default preset crosses three, four, and five constraints with easy and mixed strata; the hard preset uses three, six, and eight constraints and includes every hard constraint in each task. C22 scores only this model-visible stack and claims no separate semantic task grading.

@verify(UNIQUE)
class Preset(StrEnum):
    DEFAULT = "default"
    HARD = "hard"

PROBES: ProbePair

def score(gold: str, response: str) -> int: ...

def generate_pool(preset: Preset = Preset.DEFAULT) -> TaskPool: ...
def load_manifest(preset: Preset = Preset.DEFAULT) -> Manifest: ...

C23 subregular induction

whetstone_envs.c23 is a higher-layer environment built on the shared harness. It generates four balanced single-rule strata: ISL k=2, left-OSL k=2, right-OSL k=2, and ISL k=3 over the fixed vocabulary abcd; each task has six demonstrations and a distinct nontrivial query whose output is determinate across the complete supported hypothesis class.

Internally, the stable rule vocabulary is represented by:

@verify(UNIQUE)
class RuleFamily(StrEnum):
    ISL = "ISL"
    L_OSL = "L-OSL"
    R_OSL = "R-OSL"

@dataclass(frozen=True, slots=True)
class RuleConfiguration:
    family: RuleFamily
    context_length: int

GENERATOR_VERSION: str
PROBES: ProbePair

def generate_pool(*, n_per_stratum: int = 50) -> TaskPool: ...
def default_split_sizes(pool: TaskPool) -> tuple[int, int, int]: ...

def score_gold(prediction: str, gold: str) -> int: ...

Generation uses fixed fresh stratum seeds beginning at 555000000 and private injected random-number generators. The adapted InductionBench reference transducers and generation path are pinned and attributed inside the package; no process-global random state is read or mutated.

Terms and contracts

The published terms and contracts render the authoritative vocabulary and binding contracts directly from their TOML sources. The changelog records notable changes.

Development

Install the locked development environment and commit hook once per clone:

uv sync --locked --extra c18
uv run pre-commit install

The hook runs formatting, lint, type, definitions, the fast test suite, and package validation. Run it directly at any time:

scripts/pre-check.sh

CI also runs the full-cohort integration checks. Run that exact gate locally before release:

CI=true scripts/pre-check.sh

Regenerate the canonical C11 manifest after an intentional generator change:

uv run python -m whetstone_envs.c11.regenerate

Regenerate the canonical C19 manifest after an intentional generator change:

uv run python -m whetstone_envs.c19.regenerate