sscu-budapest utilities for scientific data engineering
Project description
datazimmer
To create a new project
- make sure that
python
points topython>=3.8
and you havepip
andgit
thenpip install datazimmer
- run
dz init project-name
- pulls project-template
- add a remote
- both to git and dvc (can run
dz build-meta
to see available dvc remotes) - git remote can be given with
dz init
- both to git and dvc (can run
- create, register and document steps in a pipeline you will run in different environments
- build metadata to exportable and serialized format with
dz build-meta
- if you defined importable data from other artifacts in the config, you can import them with
load-external-data
- ensure that you import envs that are served from sources you have access to
- if you defined importable data from other artifacts in the config, you can import them with
- build and run pipeline steps by running
dz run
- validate that the data matches the datascript description with
dz validate
Scheduling
- a project as a whole has a cron expression in
zimmer.yaml
to determine the schedule of reruns - additionally, aswan projects within the dz project can have different cron expressions for scheduling new runs of the aswan projects
Test projects
TODO: document dogshow and everything else much better here
Lookahead
- dvc style caching for explorer
- maybe separate explorer generation altogether
- overlapping names convention
- resolve naming confusion with colassigner, colaccessor and table feature / composite type / index base classes
- abstract composite type + subclass of entity class
- import ACT, inherit from it and specify
- importing composite type is impossible now if it contains foreign key :(
- add option to infer data type of assigned feature
- can be problematic b/c pandas int/float/nan issue
- create similar sets of features in a dry way
- overlapping in entities
- detect / signal the same type of entity
- exports: postgres, postgis , superset
W3C compliancy plan
- test suite for compliance: https://w3c.github.io/csvw/publishing-snapshots/PR-earl/earl.html
- https://github.com/w3c/csvw
@article{tennison2015model,
title={Model for tabular data and metadata on the web},
author={Tennison, Jeni and Kellogg, Gregg and Herman, Ivan},
year={2015}
}
@article{pollock2015metadata,
title={Metadata vocabulary for tabular data},
author={Pollock, Rufus and Tennison, Jeni and Kellogg, Gregg and Herman, Ivan},
journal={W3C Recommendation},
volume={17},
year={2015}
}
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
datazimmer-0.4.13.tar.gz
(68.1 kB
view hashes)
Built Distribution
Close
Hashes for datazimmer-0.4.13-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6226eb62e5ef2f1b7035dbf0980f58445a66de56fb9dac60ed6becce87fa19c3 |
|
MD5 | d126f8a3a06391e3dee77751edd4a8f9 |
|
BLAKE2b-256 | 5f8e42e2f03d380a19c51c17185a77b1143c4b6afb5d86d189a5a4908afa171f |