euler-dataset-contract

One small, versioned metadata boundary for Euler datasets.

euler-dataset-contract defines and validates the shared dataset-head.json format used across the Euler data tooling. It keeps dataset identity, modality semantics, and package-specific extensions in one dependency-free Python package instead of duplicating those rules in every producer and consumer.

flowchart LR
    C["ds-crawler<br/><i>produces and persists</i>"] --> H["dataset-head.json<br/><b>euler-dataset-contract</b>"]
    H --> L["euler-loading<br/><i>selects loaders and exposes metadata</i>"]
    L --> E["euler-eval<br/><i>consumes loaded values</i>"]
    H --> X["other consumers<br/><i>register their addon validators</i>"]

The package intentionally does not crawl files, load arrays, or import NumPy or PyTorch. Its core install has no runtime dependencies.

Install

Phase 1 adds opt-in modality lookups, representation/source/calibration models, planned resize/crop descriptors, generated addon schemas, and canonical hashes. See the wire format, public APIs, and compatibility rules. Core Contract 1.0 still requires modality.key; the package has no runtime dependencies. Loading's paired implementation requires these source changes.

pip install euler-dataset-contract

Python 3.9 or newer is required.

Quick start

A dataset head describes one logical dataset modality:

from euler_dataset_contract import parse_dataset_head

head = {
    "contract": {"kind": "dataset_head", "version": "1.0"},
    "dataset": {
        "id": "foggy_depth",
        "name": "Foggy Drive Depth",
        "attributes": {"split_origin": "capture_2026"},
    },
    "modality": {
        "key": "depth",
        "meta": {
            "radial_depth": False,
            "scale_to_meters": 0.001,
            "range": [0, 80],
            "file_types": ["png"],
        },
    },
    "addons": {
        "euler_loading": {
            "version": "1.0",
            "loader": "generic_dense_depth",
            "function": "depth",
        }
    },
}

contract = parse_dataset_head(head)
contract.dataset_id                    # "foggy_depth"
contract.modality_key                  # "depth"
contract.meta["scale_to_meters"]       # 0.001
contract.get_addon("euler_loading")    # package-owned extension payload
contract.to_mapping()                  # canonical, detached mapping

Invalid structural fields, missing modality metadata, malformed versions, and registered addon violations raise ValueError with a field-qualified message. Unknown modality keys remain valid and may carry arbitrary metadata, allowing the ecosystem to add modalities without first releasing this package.

Extensions

The shared contract requires every addon payload to be an object with its own version. A consuming package can register the rest of its validation rules:

from euler_dataset_contract import (
    register_addon_validator,
    validate_slot,
)

def validate_training_addon(value, context):
    validate_slot(value.get("slot"), f"{context}.slot")

register_addon_validator("euler_train", validate_training_addon)

Addon validators are process-wide. Register them during package import or application startup, and pass overwrite=True only when intentionally replacing an existing validator.

JSON Schema

Generate Draft 2020-12 schemas from the same registry used by runtime validation:

from euler_dataset_contract import (
    build_dataset_head_schema,
    build_meta_schema,
)

dataset_head_schema = build_dataset_head_schema()
modality_catalog_schema = build_meta_schema()

The repository also publishes generated schemas under schemas/. build_dataset_head_schema() selects the appropriate metadata rules from the sibling modality.key; custom modality keys accept optional custom metadata.

Built-in modality metadata

All registered modalities may additionally use the shared dimensions and file_types fields. build_default_meta(key) returns authoring defaults; parsing never silently fills missing required fields. Producer tooling such as ds-crawler may merge defaults before it validates and writes a head.

See the contract reference for field semantics, normalization, versioning, and the public API.

Conformance fixtures

The fixtures/ directory holds the shared ecosystem corpus: valid dataset heads with their canonical parsed form, and invalid heads with the error each must produce. They are JSON, so non-Python consumers read them directly.

Python consumers install the testing extra and get them as pytest fixtures through an entry point, with no conftest.py:

pip install "euler-dataset-contract[testing]"

def test_contract_heads_parse(golden_head, assert_head_roundtrip):
    assert_head_roundtrip(golden_head.head)

That single function runs once per head in the corpus. The extra carries only pytest; the core install stays dependency-free and never imports it.

See Conformance fixtures for the full fixture list, the shipped modality inventory, and the rule that a contract change is proven against the corpus in every repository before it ships.

Phase 0 evidence adds the current dataset vocabulary, 90 loader declarations, shared resize/crop and projection examples, and opt-in checks against ds-crawler, loading, preprocess, and eval. The report separates reference results from reproduced legacy failures and includes a local test driver. Contract 1.0 runtime behavior is unchanged.

Ecosystem boundary

The current 1.0 contract describes persisted dataset metadata. It does not yet guarantee the exact decoded tensor layout, dtype, coordinate frame, or value domain returned by a loader. In practice, euler-loading owns those decoding rules and downstream packages normalize its outputs.

The observed gaps and a staged, backwards-compatible route toward canonical modality names and enforceable decoded-output contracts are documented in Future contract directions. The Ecosystem streamlining plan turns that direction into a concrete design and rollout.

Development

git clone https://github.com/d-rothen/euler-dataset-contract.git
cd euler-dataset-contract
uv sync --extra dev
uv run ruff check .
uv run pytest
uv build
uv run python scripts/verify_distribution.py dist/*

See CONTRIBUTING.md for schema changes and release checks, and SECURITY.md for private vulnerability reporting.

License

MIT © Daniel Rothenpieler

Phase 2 adds opt-in captured spatial exports and validated materialization, with verification and source installation details.