Skip to main content

Tests Coverage Status Binder

Welcome to zdatasets

Development

  • Set the version to a dev version, e.g. 1.3.0.dev1 in pyproject.toml when starting development.
  • Bump the dev version (e.g., 1.3.0.dev1 → 1.3.0.dev2) every time you have a change you want to test in other repositories.
  • After every change, confirm that the github workflow runs are successful at https://github.com/zillow/zdatasets/actions.
  • The dev versions are published in test PyPI at https://test.pypi.org/project/zdatasets/#history.
  • While testing your changes, you may need to reference your merge request in other repositories' pyproject.toml instead of using the dev version. For example,
dataset = [
  "zdatasets[kubernetes] @ git+https://github.com/zillow/zdatasets.git@refs/pull/42/head"
]

Example

import pandas as pd
from metaflow import FlowSpec, step

from zdatasets import Dataset, Mode
from zdatasets.metaflow import DatasetParameter
from zdatasets.plugins import BatchOptions


# Can also invoke from CLI:
#  > python zdatasets/tutorials/0_hello_dataset_flow.py run \
#    --hello_dataset '{"name": "HelloDataset", "mode": "READ_WRITE", \
#    "options": {"type": "BatchOptions", "partition_by": "region"}}'
class HelloDatasetFlow(FlowSpec):
    hello_dataset = DatasetParameter(
        "hello_dataset",
        default=Dataset("HelloDataset", mode=Mode.READ_WRITE, options=BatchOptions(partition_by="region")),
    )

    @step
    def start(self):
        df = pd.DataFrame({"region": ["A", "A", "A", "B", "B", "B"], "zpid": [1, 2, 3, 4, 5, 6]})
        print("saving data_frame: \n", df.to_string(index=False))

        # Example of writing to a dataset
        self.hello_dataset.write(df)

        # save this as an output dataset
        self.output_dataset = self.hello_dataset

        self.next(self.end)

    @step
    def end(self):
        print(f"I have dataset \n{self.output_dataset=}")

        # output_dataset to_pandas(partitions=dict(region="A")) only
        df: pd.DataFrame = self.output_dataset.to_pandas(partitions=dict(region="A"))
        print('self.output_dataset.to_pandas(partitions=dict(region="A")):')
        print(df.to_string(index=False))


if __name__ == "__main__":
    HelloDatasetFlow()

Metadata

Release files for zdatasets 1.3.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for zdatasets 1.3.0
File Size Uploaded Provenance Actions
zdatasets-1.3.0.tar.gz 55.3 kB Without provenance Details for zdatasets-1.3.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for zdatasets 1.3.0
File Interpreter ABI Platform Actions
zdatasets-1.3.0-py3-none-any.whl Python 3 none any Details for zdatasets-1.3.0-py3-none-any.whl

Total release size: 141.4 kB

Release files / zdatasets-1.3.0.tar.gz

Download URL zdatasets-1.3.0.tar.gz
Size 55.3 kB
Tags Source
SHA-256 checksum
How to use checksums
a5376c92a53d28a96832b2ecffb652d52b0c08f6966f97dfeb070ef62984b476
BLAKE2b-256 checksum
How to use checksums
fafaa4e4a63d421909eeae8659e90383a67e06b216234b5c59f5fd1927254881
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.13.7

Release files / zdatasets-1.3.0-py3-none-any.whl

Download URL zdatasets-1.3.0-py3-none-any.whl
Size 86.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
d66905334358b8f7d2ea30b5853ee1ba1627891ca1281fbbd3dcc3f22e892abd
BLAKE2b-256 checksum
How to use checksums
3f6dd654a34ca9225c0e8cdb00918cecf51ff9e1b930dac815693039501bfe77
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.13.7

Security for zdatasets 1.3.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page