Skip to main content

euler

euler-dataset-contract

One small, versioned metadata boundary for Euler datasets.

PyPI Python versions License: MIT CI


euler-dataset-contract defines and validates the shared dataset-head.json format used across the Euler data tooling. It keeps dataset identity, modality semantics, and package-specific extensions in one dependency-free Python package instead of duplicating those rules in every producer and consumer.

flowchart LR
    C["ds-crawler<br/><i>produces and persists</i>"] --> H["dataset-head.json<br/><b>euler-dataset-contract</b>"]
    H --> L["euler-loading<br/><i>selects loaders and exposes metadata</i>"]
    L --> E["euler-eval<br/><i>consumes loaded values</i>"]
    H --> X["other consumers<br/><i>register their addon validators</i>"]

The package intentionally does not crawl files, load arrays, or import NumPy or PyTorch. Its core install has no runtime dependencies.

Install

Phase 1 adds opt-in modality lookups, representation/source/calibration models, planned resize/crop descriptors, generated addon schemas, and canonical hashes. See the wire format, public APIs, and compatibility rules. Core Contract 1.0 still requires modality.key; the package has no runtime dependencies. Loading's paired implementation requires these source changes.

pip install euler-dataset-contract

Python 3.9 or newer is required.

Quick start

A dataset head describes one logical dataset modality:

from euler_dataset_contract import parse_dataset_head

head = {
    "contract": {"kind": "dataset_head", "version": "1.0"},
    "dataset": {
        "id": "foggy_depth",
        "name": "Foggy Drive Depth",
        "attributes": {"split_origin": "capture_2026"},
    },
    "modality": {
        "key": "depth",
        "meta": {
            "radial_depth": False,
            "scale_to_meters": 0.001,
            "range": [0, 80],
            "file_types": ["png"],
        },
    },
    "addons": {
        "euler_loading": {
            "version": "1.0",
            "loader": "generic_dense_depth",
            "function": "depth",
        }
    },
}

contract = parse_dataset_head(head)
contract.dataset_id                    # "foggy_depth"
contract.modality_key                  # "depth"
contract.meta["scale_to_meters"]       # 0.001
contract.get_addon("euler_loading")    # package-owned extension payload
contract.to_mapping()                  # canonical, detached mapping

Invalid structural fields, missing modality metadata, malformed versions, and registered addon violations raise ValueError with a field-qualified message. Unknown modality keys remain valid and may carry arbitrary metadata, allowing the ecosystem to add modalities without first releasing this package.

Extensions

The shared contract requires every addon payload to be an object with its own version. A consuming package can register the rest of its validation rules:

from euler_dataset_contract import (
    register_addon_validator,
    validate_slot,
)

def validate_training_addon(value, context):
    validate_slot(value.get("slot"), f"{context}.slot")

register_addon_validator("euler_train", validate_training_addon)

Addon validators are process-wide. Register them during package import or application startup, and pass overwrite=True only when intentionally replacing an existing validator.

JSON Schema

Generate Draft 2020-12 schemas from the same registry used by runtime validation:

from euler_dataset_contract import (
    build_dataset_head_schema,
    build_meta_schema,
)

dataset_head_schema = build_dataset_head_schema()
modality_catalog_schema = build_meta_schema()

The repository also publishes generated schemas under schemas/. build_dataset_head_schema() selects the appropriate metadata rules from the sibling modality.key; custom modality keys accept optional custom metadata.

Built-in modality metadata

Modality key Required metadata
rgb range
depth radial_depth, scale_to_meters, range
segmentation skyclass
semantic_segmentation skyclass

All registered modalities may additionally use the shared dimensions and file_types fields. build_default_meta(key) returns authoring defaults; parsing never silently fills missing required fields. Producer tooling such as ds-crawler may merge defaults before it validates and writes a head.

See the contract reference for field semantics, normalization, versioning, and the public API.

Conformance fixtures

The fixtures/ directory holds the shared ecosystem corpus: valid dataset heads with their canonical parsed form, and invalid heads with the error each must produce. They are JSON, so non-Python consumers read them directly.

Python consumers install the testing extra and get them as pytest fixtures through an entry point, with no conftest.py:

pip install "euler-dataset-contract[testing]"
def test_contract_heads_parse(golden_head, assert_head_roundtrip):
    assert_head_roundtrip(golden_head.head)

That single function runs once per head in the corpus. The extra carries only pytest; the core install stays dependency-free and never imports it.

See Conformance fixtures for the full fixture list, the shipped modality inventory, and the rule that a contract change is proven against the corpus in every repository before it ships.

Phase 0 evidence adds the current dataset vocabulary, 90 loader declarations, shared resize/crop and projection examples, and opt-in checks against ds-crawler, loading, preprocess, and eval. The report separates reference results from reproduced legacy failures and includes a local test driver. Contract 1.0 runtime behavior is unchanged.

Ecosystem boundary

The current 1.0 contract describes persisted dataset metadata. It does not yet guarantee the exact decoded tensor layout, dtype, coordinate frame, or value domain returned by a loader. In practice, euler-loading owns those decoding rules and downstream packages normalize its outputs.

The observed gaps and a staged, backwards-compatible route toward canonical modality names and enforceable decoded-output contracts are documented in Future contract directions. The Ecosystem streamlining plan turns that direction into a concrete design and rollout.

Development

git clone https://github.com/d-rothen/euler-dataset-contract.git
cd euler-dataset-contract
uv sync --extra dev
uv run ruff check .
uv run pytest
uv build
uv run python scripts/verify_distribution.py dist/*

See CONTRIBUTING.md for schema changes and release checks, and SECURITY.md for private vulnerability reporting.

License

MIT © Daniel Rothenpieler

Phase 2 adds opt-in captured spatial exports and validated materialization, with verification and source installation details.

Metadata

Release files for euler-dataset-contract 0.11.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for euler-dataset-contract 0.11.0
File Size Uploaded Provenance Actions
euler_dataset_contract-0.11.0.tar.gz 205.5 kB Without provenance Details for euler_dataset_contract-0.11.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for euler-dataset-contract 0.11.0
File Interpreter ABI Platform Actions
euler_dataset_contract-0.11.0-py3-none-any.whl Python 3 none any Details for euler_dataset_contract-0.11.0-py3-none-any.whl

Total release size: 370.5 kB

Release files / euler_dataset_contract-0.11.0.tar.gz

Download URL euler_dataset_contract-0.11.0.tar.gz
Size 205.5 kB
Tags Source
SHA-256 checksum
How to use checksums
e79301cb26077e1cd22ee126e8fa5e3ee17ad48233e96b436f485aab124bd5e0
BLAKE2b-256 checksum
How to use checksums
9e7d2a9a3239b299e682153c0112427ac5e34ff4065352f99a48f76dd5d71773
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / euler_dataset_contract-0.11.0-py3-none-any.whl

Download URL euler_dataset_contract-0.11.0-py3-none-any.whl
Size 164.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
fb41477bbc5dc53fc7171a7f4b6a6b6bea3849ee192f15960303d87090553acd
BLAKE2b-256 checksum
How to use checksums
c47533d2e1c8606f2fd6df65a21d13aea20a2a9a00f8ba586112c444c09a5c55
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for euler-dataset-contract 0.11.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page