whetstone-envs

Task-family-agnostic contracts for reproducible quick-test environments.

Scope

This repo owns the environment data and evaluation rules shared by Whetstone's quick-test task families, with no dependency on optimizer or execution-contract code:

• Instances define immutable task inputs, private gold data, generation seeds, task strata, and public prompt identity.
• Pools and splits validate ordered instance collections and allocate deterministic internal, official, and held-out cohorts.
• Probes pair naive and ceiling templates, render public prompt inputs, and normalize predictions for evaluation.
• Scoring represents scored, failed, and missing observations and aggregates complete repeat matrices through task, stratum, and overall levels.
• Manifests pin generated pools with versioned identities and bounded canonical persistence.

The task-family implementations and the adapter to Whetstone's optimizer live above this shared harness rather than inside its contracts.

Installation

uv add whetstone-envs

Instances

whetstone_envs.instances owns the immutable unit passed through generation, prompting, scoring, splitting, and persistence. Prompt inputs are public; gold remains private evaluation data.

@dataclass(frozen=True, slots=True)
class Instance:
    id: str
    seed: int
    strata: tuple[str, ...]
    prompt_inputs: Mapping[str, str] = field(default_factory=lambda: ...)
    gold: str = ""

def make_instance(
    *,
    id: str,
    seed: int,
    strata: tuple[str, ...] | str,
    prompt_inputs: Mapping[str, str] | None = None,
    gold: str = "",
) -> Instance: ...

def public_prompt_identity(
    instance: Instance,
) -> tuple[tuple[str, str], ...]: ...

Pools and splits

whetstone_envs.pools owns validated ordered pools and the deterministic policy for selecting three disjoint evaluation cohorts. Split optimization is delegated to dr-graph; returned instances preserve pool order.

@dataclass(frozen=True, slots=True)
class PoolSplit:
    internal_eval: tuple[Instance, ...]
    official: tuple[Instance, ...]
    held_out: tuple[Instance, ...]

@dataclass(frozen=True, slots=True)
class TaskPool:
    instances: tuple[Instance, ...]

    @property
    def strata(self) -> tuple[str, ...]: ...

    def stratum_counts(self) -> dict[str, int]: ...
    def in_stratum(self, label: str) -> tuple[Instance, ...]: ...
    def split(
        self,
        internal_eval_n: int,
        official_n: int,
        held_out_n: int,
    ) -> PoolSplit: ...

Probes

whetstone_envs.probes owns the floor/ceiling prompt pair and the default renderer that can see only public prompt inputs. Normalization strips whitespace and complete outer triple-backtick fences.

def render_with_prompt_inputs(template: str, instance: Instance) -> str: ...
def normalize(prediction: str) -> str: ...

@dataclass(frozen=True, slots=True)
class ProbePair:
    naive_template: str
    ceiling_template: str
    render: Callable[[str, Instance], str] = render_with_prompt_inputs

    def render_naive(self, instance: Instance) -> str: ...
    def render_ceiling(self, instance: Instance) -> str: ...

Scoring

whetstone_envs.scoring keeps failures and absent results distinct from binary scores. Aggregation exposes a mean only when the complete planned task/repeat matrix is present and scored.

@verify(UNIQUE)
class Outcome(StrEnum):
    SCORED = "scored"
    FAILED = "failed"
    MISSING = "missing"

@dataclass(frozen=True, slots=True)
class Observation:
    task_id: str
    repeat_id: int
    outcome: Outcome = Outcome.SCORED
    score: int | None = None

@dataclass(frozen=True, slots=True)
class Aggregate:
    mean: float | None
    usable: int
    failed_count: int
    missing_count: int
    label: str | None = None
    children: tuple["Aggregate", ...] = field(default_factory=tuple)

def aggregate(
    observations: Iterable[Observation],
    task_strata: Mapping[str, tuple[str, ...]],
    *,
    expected_repeat_ids: Iterable[int],
) -> Aggregate: ...

exact_match, scored, failed, and missing provide the primary leaf-level constructors. aggregate_task, aggregate_stratum, and aggregate_overall expose the individual aggregation steps when callers already own the hierarchy.

Manifests

whetstone_envs.manifests owns the serialized boundary for regenerated pool identity. Manifests use a closed Pydantic schema, dr-serialize identities, and dr-store canonical files.

class Manifest(BaseModel):
    generator_version: str
    seed_range: tuple[int, int]
    stratum_counts: Mapping[str, int]
    content_hash: Sha256Digest
    schema_version: int = MANIFEST_SCHEMA_VERSION

    @classmethod
    def from_pool(
        cls,
        pool: TaskPool,
        *,
        generator_version: str,
        seed_range: tuple[int, int],
    ) -> "Manifest": ...

    def write(self, path: Path) -> None: ...
    @classmethod
    def read(cls, path: Path) -> "Manifest": ...
    def matches_pool(self, pool: TaskPool) -> bool: ...

def content_hash(pool: TaskPool) -> Sha256Digest: ...

Terms and contracts

The published terms and contracts render the authoritative vocabulary and binding contracts directly from their TOML sources. The changelog records notable changes.

Development

Install the locked development environment and commit hook once per clone:

uv sync --locked
uv run pre-commit install

The hook runs the same formatting, lint, type, definitions, test, and package build gate used by CI. Run it directly at any time:

scripts/pre-check.sh