Skip to main content

euler

euler-dataset-contract

One small, versioned metadata boundary for Euler datasets.

PyPI Python versions License: MIT CI


euler-dataset-contract defines and validates the shared dataset-head.json format used across the Euler data tooling. It keeps dataset identity, modality semantics, and package-specific extensions in one dependency-free Python package instead of duplicating those rules in every producer and consumer.

flowchart LR
    C["ds-crawler<br/><i>produces and persists</i>"] --> H["dataset-head.json<br/><b>euler-dataset-contract</b>"]
    H --> L["euler-loading<br/><i>selects loaders and exposes metadata</i>"]
    L --> E["euler-eval<br/><i>consumes loaded values</i>"]
    H --> X["other consumers<br/><i>register their addon validators</i>"]

The package intentionally does not crawl files, load arrays, or import NumPy or PyTorch. Its core install has no runtime dependencies.

Install

Phase 1 adds opt-in modality lookups, representation/source/calibration models, planned resize/crop descriptors, generated addon schemas, and canonical hashes. See the wire format, public APIs, and compatibility rules. Core Contract 1.0 still requires modality.key; the package has no runtime dependencies. Loading's paired implementation requires these source changes.

pip install euler-dataset-contract

Python 3.9 or newer is required.

Quick start

A dataset head describes one logical dataset modality:

from euler_dataset_contract import parse_dataset_head

head = {
    "contract": {"kind": "dataset_head", "version": "1.0"},
    "dataset": {
        "id": "foggy_depth",
        "name": "Foggy Drive Depth",
        "attributes": {"split_origin": "capture_2026"},
    },
    "modality": {
        "key": "depth",
        "meta": {
            "radial_depth": False,
            "scale_to_meters": 0.001,
            "range": [0, 80],
            "file_types": ["png"],
        },
    },
    "addons": {
        "euler_loading": {
            "version": "1.0",
            "loader": "generic_dense_depth",
            "function": "depth",
        }
    },
}

contract = parse_dataset_head(head)
contract.dataset_id                    # "foggy_depth"
contract.modality_key                  # "depth"
contract.meta["scale_to_meters"]       # 0.001
contract.get_addon("euler_loading")    # package-owned extension payload
contract.to_mapping()                  # canonical, detached mapping

Invalid structural fields, missing modality metadata, malformed versions, and registered addon violations raise ValueError with a field-qualified message. Unknown modality keys remain valid and may carry arbitrary metadata, allowing the ecosystem to add modalities without first releasing this package.

Extensions

The shared contract requires every addon payload to be an object with its own version. A consuming package can register the rest of its validation rules:

from euler_dataset_contract import (
    register_addon_validator,
    validate_slot,
)

def validate_training_addon(value, context):
    validate_slot(value.get("slot"), f"{context}.slot")

register_addon_validator("euler_train", validate_training_addon)

Addon validators are process-wide. Register them during package import or application startup, and pass overwrite=True only when intentionally replacing an existing validator.

The ecosystem's own euler_loading addon, which names a dataset's decoder, ships its rules here. Call register_euler_loading_addon_validator() to apply them; ds-crawler and euler-loading do so on import. See the contract reference.

JSON Schema

Generate Draft 2020-12 schemas from the same registry used by runtime validation:

from euler_dataset_contract import (
    build_dataset_head_schema,
    build_meta_schema,
)

dataset_head_schema = build_dataset_head_schema()
modality_catalog_schema = build_meta_schema()

The repository also publishes generated schemas under schemas/. build_dataset_head_schema() selects the appropriate metadata rules from the sibling modality.key; custom modality keys accept optional custom metadata.

Built-in modality metadata

Modality key Required metadata
rgb range
depth radial_depth, scale_to_meters, range
segmentation skyclass
semantic_segmentation skyclass

All registered modalities may additionally use the shared dimensions, file_types, invalid, sparse and source_grid fields. invalid says how pixels without a valid value are marked (a sentinel, non-finite values); sparse marks rasters such as lidar depth that hold measurements at only some pixels; source_grid records how a dataset's pixel grid was cropped or resized from the images it was computed from (see the contract reference). build_default_meta(key) returns authoring defaults; parsing never silently fills missing required fields. Producer tooling such as ds-crawler may merge defaults before it validates and writes a head.

See the contract reference for field semantics, normalization, versioning, and the public API.

Conformance fixtures

The fixtures/ directory holds the shared ecosystem corpus: valid dataset heads with their canonical parsed form, and invalid heads with the error each must produce. They are JSON, so non-Python consumers read them directly.

Python consumers install the testing extra and get them as pytest fixtures through an entry point, with no conftest.py:

pip install "euler-dataset-contract[testing]"
def test_contract_heads_parse(golden_head, assert_head_roundtrip):
    assert_head_roundtrip(golden_head.head)

That single function runs once per head in the corpus. The extra carries only pytest; the core install stays dependency-free and never imports it.

See Conformance fixtures for the full fixture list, the shipped modality inventory, and the rule that a contract change is proven against the corpus in every repository before it ships.

Phase 0 evidence adds the current dataset vocabulary, 90 loader declarations, shared resize/crop and projection examples, and opt-in checks against ds-crawler, loading, preprocess, and eval. The report separates reference results from reproduced legacy failures and includes a local test driver. Contract 1.0 runtime behavior is unchanged.

Ecosystem boundary

The current 1.0 contract describes persisted dataset metadata. It does not yet guarantee the exact decoded tensor layout, dtype, coordinate frame, or value domain returned by a loader. In practice, euler-loading owns those decoding rules and downstream packages normalize its outputs.

The observed gaps and a staged, backwards-compatible route toward canonical modality names and enforceable decoded-output contracts are documented in Future contract directions. The Ecosystem streamlining plan turns that direction into a concrete design and rollout.

Development

git clone https://github.com/d-rothen/euler-dataset-contract.git
cd euler-dataset-contract
uv sync --extra dev
uv run ruff check .
uv run pytest
uv build
uv run python scripts/verify_distribution.py dist/*

See CONTRIBUTING.md for schema changes and release checks, and SECURITY.md for private vulnerability reporting.

License

MIT © Daniel Rothenpieler

Phase 2 adds opt-in captured spatial exports and validated materialization, with verification and source installation details.

Metadata

Release files for euler-dataset-contract 0.14.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for euler-dataset-contract 0.14.0
File Size Uploaded Provenance Actions
euler_dataset_contract-0.14.0.tar.gz 222.6 kB Without provenance Details for euler_dataset_contract-0.14.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for euler-dataset-contract 0.14.0
File Interpreter ABI Platform Actions
euler_dataset_contract-0.14.0-py3-none-any.whl Python 3 none any Details for euler_dataset_contract-0.14.0-py3-none-any.whl

Total release size: 404.0 kB

Release files / euler_dataset_contract-0.14.0.tar.gz

Download URL euler_dataset_contract-0.14.0.tar.gz
Size 222.6 kB
Tags Source
SHA-256 checksum
How to use checksums
e23bbae8c4226600225352211c33c82264d43dd0e5159c7c71b8b97ca1162701
BLAKE2b-256 checksum
How to use checksums
b098bc1ae25ea4512f87dca9ef829170729636f3f0c64972228a3f4c7322a658
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / euler_dataset_contract-0.14.0-py3-none-any.whl

Download URL euler_dataset_contract-0.14.0-py3-none-any.whl
Size 181.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
72c868cf6bb7e7c39c11a1acce878e145019bf6240333f6f7779b3b87ac2d635
BLAKE2b-256 checksum
How to use checksums
2345ccdab315dc6e7731b018f55a8d3ef0840616ff667a41704b3b7a5db7cbf2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for euler-dataset-contract 0.14.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page