Skip to main content

Dagster library for ser/de of data

Project description

dagster-serde

image image image CI pre-commit Checked with pyright Ruff Code style: black

Type-aware ser/de library for Dagster.

@dataclass
class MyStruct:
    foo: str


my_struct = MyStruct(foo="bar")


@asset(
    io_manager_key="json_io_manager",
)
def upstream() -> MyStruct:
    return my_struct  # my_struct is serialized to a json file (type hint is optional)


@asset
def downstream(
    upstream: MyStruct,
):  # my_struct is loaded as MyStruct based on the type hint
    assert upstream == my_struct

Installation

pip install dagster-serde

List of available extras:

  • json
  • yaml

For example, to use the JsonIOManager, install dagster-serde[json].

IOManagers

BaseSerdeUPathIOManager

Base class for IOManagers that ser/de to/from filesystems.

  • Automatically deserializes inputs based on type annotations. If the input annotation is not a supported type, it's returned as a normal built-in type (e.g. str, int, dict, etc.). The supported types are:

    • dataclasses.dataclass and serde.serde (using pyserde). The full list of supported types can be found here.
    • TODO: add support for other libraries like pydantic
  • Optional type annotations are supported. If the input annotation is Optional and is missing in the filesystem, the IOManager will skip loading the input and return None instead. If the output annotation is Optional and the output is None, the IOManager will skip writing the output to the filesystem.

JsonIOManager

Implements BaseSerdeUPathIOManager for JSON files. Uses orjson.

YamlIOManager

Implements BaseSerdeUPathIOManager for YAML files. Uses pyyaml.

Examples

See examples.

Development

Installation

poetry install
poetry run pre-commit install

Testing

poetry run pytest

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dagster_serde-0.0.2.tar.gz (8.7 kB view details)

Uploaded Source

Built Distribution

dagster_serde-0.0.2-py3-none-any.whl (9.9 kB view details)

Uploaded Python 3

File details

Details for the file dagster_serde-0.0.2.tar.gz.

File metadata

  • Download URL: dagster_serde-0.0.2.tar.gz
  • Upload date:
  • Size: 8.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.4.2 CPython/3.9.18 Linux/6.2.0-1012-azure

File hashes

Hashes for dagster_serde-0.0.2.tar.gz
Algorithm Hash digest
SHA256 b63b78b122d327e5f6b1fa00299a91d4f9c2ba500040695fb93765696f38124b
MD5 caa2b251038b7873d56a3b196a4fb7a4
BLAKE2b-256 b41581f987e396f88c6121b6ca700a8157c9eb70366c23d8d3b57ffd011b1b55

See more details on using hashes here.

File details

Details for the file dagster_serde-0.0.2-py3-none-any.whl.

File metadata

  • Download URL: dagster_serde-0.0.2-py3-none-any.whl
  • Upload date:
  • Size: 9.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.4.2 CPython/3.9.18 Linux/6.2.0-1012-azure

File hashes

Hashes for dagster_serde-0.0.2-py3-none-any.whl
Algorithm Hash digest
SHA256 6cded910845e4fdb0e6f1fdf80cc98dff167b83589058c39cde457d9f7bd87ac
MD5 d51da8c7a6995a28bce8a881d3d0062f
BLAKE2b-256 04846346827dc2c4696ebc52af6a3d1e0aab18842bdb4c5ed33042e8887d7db6

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page