Skip to main content

datazimmer

pypi DOI

Scientific data pipeline orchestrator. Define typed entity schemas, build reproducible polars pipelines, validate foreign-key integrity, and publish versioned datasets consumed by URL.

Part of the sscu-budapest ecosystem for open scientific data.

Install

uv add datazimmer
uv add "datazimmer[collect]"   # + aswan web collection
uv add "datazimmer[s3]"        # + S3 publishing/consumption
uv add "datazimmer[zenodo]"    # + Zenodo deposition

Concepts

A project is one importable package exposing a module-level pipeline, configured by a [tool.datazimmer] table in pyproject.toml. Everything is plain classes, type hints and polars frames.

Entities — typed schemas

from typing import Annotated, Literal
from datazimmer import Entity, Index

class Breed(Entity):
    breed_id: Annotated[str, Index]   # index = unique key, sort order
    name: str
    origin: str | None                # nullable

class Dog(Entity):
    dog_id: Annotated[str, Index]
    name: str
    breed: Breed                      # foreign key -> breed__breed_id
    weight_kg: float
    sex: Literal["m", "f"]            # categorical -> pl.Enum

The index and the foreign keys drive SQL materialization, ER diagrams and published manifests.

Tables — schema-validated parquet (via shackleton)

from datazimmer import Table
breeds = Table(Breed, "data/breeds")
breeds.write(df)          # validated + dtype-coerced
breeds.replace(df)        # upsert on the index, composite or not
lf = breeds.read()        # polars LazyFrame

Files stay sorted by the entity's index.

Pipeline — explicit steps, hash-based skipping

from datazimmer import Pipeline
pipeline = Pipeline()

@pipeline.step(outputs=[breeds, dogs])
def seed(): ...

@pipeline.step(inputs=[dogs], outputs=[breed_wins])
def aggregate(): ...

An input is a Table, a path, or another step. dz run runs the steps in dependency order and skips one whose source and inputs are unchanged since it last succeeded and whose outputs exist, so editing a step re-runs it and everything downstream. Fingerprints are kept in the project's state dir (config.STATE_DIR).

Validate, publish, consume

from datazimmer import Config, validate, publish, load, load_dependency
validate([breeds, dogs])                                  # uniqueness + FK integrity
publish(Config.load(), [breeds, dogs])                    # versioned upload + manifest
lf = load("s3://bucket/datasets/dogshow", "breeds")       # lazy scan (local/s3/http)
lf = load_dependency(Config.load(), "elections", "precincts")  # a [tool.datazimmer.dependencies] entry

A consumer needs only the URL: the manifest carries each table's schema and files. dz schema codegen turns it into typed Entity classes when you want them.

Collection — aswan collectors

from datazimmer import DzAswan

class Lichess(DzAswan):
    name = "lichess"                  # the aswan depot
    cadence = "1h"                    # count + unit of CADENCE_UNITS; unset = only when named
    starters = {GetTournaments: ["https://lichess.org/api/tournament"]}

@pipeline.step(outputs=[tournaments])
def parse():
    tournaments.write(pl.DataFrame([e.content for e in Lichess().events(GetTournaments)]))

dz collect imports the collectors modules and runs every collector whose cadence has passed since its last attempt, into the aswan depot root (ASWAN_DEPOT_ROOT); with ASWAN_REMOTE set, it pulls the depot first and pushes it after. A failed run counts as an attempt, so a broken collector retries once per cadence. Call it from a timer (cron, systemd) as often as the shortest cadence needs.

Configuration (pyproject.toml)

[tool.datazimmer]
name = "dogshow"
version = "0.1.0"
package = "src"
collectors = ["src.collection"]   # modules whose DzAswan classes `dz collect` runs

[tool.datazimmer.remote]
kind = "s3"                       # local | ssh | s3
url = "s3://bucket/datasets"

[tool.datazimmer.dependencies.elections]
source = "s3://bucket/datasets/electiondata"
version = "0.2.0"                 # omit for the latest

Credentials are the standard env vars (AWS_ACCESS_KEY_ID, ZENODO_TOKEN, …). Zenodo deposition is Python-only: datazimmer.zenodo.ZenodoDeposit creates a draft, uploads files and optionally publishes it.

CLI

dz init <name>            # scaffold a new project
dz run [--force]          # execute the pipeline, skipping unchanged steps
dz collect [names…]       # run due collectors, or the named ones [--force] [--status]
dz validate               # index uniqueness + foreign-key integrity
dz sql [--constr URL]     # load all tables into a relational database
dz draw [--out PATH]      # write a Mermaid ER diagram
dz publish                # upload tables + manifest to the remote
dz pull <source>          # download a published project into the cache
dz schema export          # JSON Schema per entity
dz schema codegen <src>   # Entity classes from a published manifest

dz --help-all prints every command's flags; dz --complete <args…> serves shell completion.

The dogshow

dogshow/ is the canonical example and the cross-package end-to-end suite: aswan scrapes a locally served kennel federation into a depot, the pipeline parses it into a typed star schema, and the result is validated, materialized to SQL + an ER diagram, and published. It reads no external data.

cd dogshow && make test                                      # self-contained suite
docker compose -f dogshow/docker/compose.yaml up --build     # + the ssh backend, combined coverage of every package

Ecosystem

Library Role
shackleton Polars table storage
aswan Web data collection
dirark Content-addressed archival (aswan depot)
atqo Task queue orchestration (aswan workers)
sqlmermaid ER diagram generation
protocli The dz CLI dispatcher

License

See LICENSE.

Metadata

Release files for datazimmer 1.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for datazimmer 1.0.0
File Size Uploaded Provenance Actions
datazimmer-1.0.0.tar.gz 31.7 kB Without provenance Details for datazimmer-1.0.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for datazimmer 1.0.0
File Interpreter ABI Platform Actions
datazimmer-1.0.0-py3-none-any.whl Python 3 none any Details for datazimmer-1.0.0-py3-none-any.whl

Total release size: 69.2 kB

Release files / datazimmer-1.0.0.tar.gz

Download URL datazimmer-1.0.0.tar.gz
Size 31.7 kB
Tags Source
SHA-256 checksum
How to use checksums
cdc141f645fe76bafbfeed2879849611983218bc0ac8f7eee5c6121a7cd5cc6f
BLAKE2b-256 checksum
How to use checksums
09410824204109b5da18b1013c427d2098c500ffbd65a66d1f5d7e893af9adb6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.3 {"installer":{"name":"uv","version":"0.13.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / datazimmer-1.0.0-py3-none-any.whl

Download URL datazimmer-1.0.0-py3-none-any.whl
Size 37.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
1196984c4001c56174e02ab0816fa1c43f00741c7cd09afb74690b174fa4f308
BLAKE2b-256 checksum
How to use checksums
d6ae424cd145cac39ffc9c1860c340fe483306a39ebc3bb8eb93a87e6bb2ee4b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.3 {"installer":{"name":"uv","version":"0.13.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for datazimmer 1.0.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.0.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page