Dagster library for ser/de of data
Project description
dagster-serde
Type-aware ser/de library for Dagster.
@dataclass
class MyStruct:
foo: str
my_struct = MyStruct(foo="bar")
@asset(
io_manager_key="json_io_manager",
)
def upstream() -> MyStruct:
return my_struct # my_struct is serialized to a json file (type hint is optional)
@asset
def downstream(
upstream: MyStruct,
): # my_struct is loaded as MyStruct based on the type hint
assert upstream == my_struct
Installation
pip install dagster-serde
List of available extras:
json
yaml
For example, to use the JsonIOManager
, install dagster-serde[json]
.
IOManagers
BaseSerdeUPathIOManager
Base class for IOManagers that ser/de to/from filesystems.
-
Automatically deserializes inputs based on type annotations. If the input annotation is not a supported type, it's returned as a normal built-in type (e.g.
str
,int
,dict
, etc.). The supported types are:dataclasses.dataclass
andserde.serde
(usingpyserde
). The full list of supported types can be found here.- TODO: add support for other libraries like
pydantic
-
Optional
type annotations are supported. If the input annotation isOptional
and is missing in the filesystem, theIOManager
will skip loading the input and returnNone
instead. If the output annotation isOptional
and the output isNone
, theIOManager
will skip writing the output to the filesystem.
JsonIOManager
Implements BaseSerdeUPathIOManager
for JSON files. Uses orjson
.
YamlIOManager
Implements BaseSerdeUPathIOManager
for YAML files. Uses pyyaml
.
Examples
See examples.
Development
Installation
poetry install
poetry run pre-commit install
Testing
poetry run pytest
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file dagster_serde-0.0.2.tar.gz
.
File metadata
- Download URL: dagster_serde-0.0.2.tar.gz
- Upload date:
- Size: 8.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.4.2 CPython/3.9.18 Linux/6.2.0-1012-azure
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | b63b78b122d327e5f6b1fa00299a91d4f9c2ba500040695fb93765696f38124b |
|
MD5 | caa2b251038b7873d56a3b196a4fb7a4 |
|
BLAKE2b-256 | b41581f987e396f88c6121b6ca700a8157c9eb70366c23d8d3b57ffd011b1b55 |
File details
Details for the file dagster_serde-0.0.2-py3-none-any.whl
.
File metadata
- Download URL: dagster_serde-0.0.2-py3-none-any.whl
- Upload date:
- Size: 9.9 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.4.2 CPython/3.9.18 Linux/6.2.0-1012-azure
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6cded910845e4fdb0e6f1fdf80cc98dff167b83589058c39cde457d9f7bd87ac |
|
MD5 | d51da8c7a6995a28bce8a881d3d0062f |
|
BLAKE2b-256 | 04846346827dc2c4696ebc52af6a3d1e0aab18842bdb4c5ed33042e8887d7db6 |