sscu-budapest utilities for scientific data engineering
Project description
datazimmer
To create a new project
- make sure that
python
points topython>=3.8
and you havepip
andgit
thenpip install datazimmer
- run
dz init project-name
- pulls project-template
- add a remote
- both to git and dvc (can run
dz build-meta
to see available dvc remotes) - git remote can be given with
dz init
- both to git and dvc (can run
- create, register and document steps in a pipeline you will run in different environments
- build metadata to exportable and serialized format with
dz build-meta
- if you defined importable data from other artifacts in the config, you can import them with
load-external-data
- ensure that you import envs that are served from sources you have access to
- if you defined importable data from other artifacts in the config, you can import them with
- build and run pipeline steps by running
dz run
- validate that the data matches the datascript description with
dz validate
Scheduling
- a project as a whole has a cron expression in
zimmer.yaml
to determine the schedule of reruns - additionally, aswan projects within the dz project can have different cron expressions for scheduling new runs of the aswan projects
Test projects
TODO: document dogshow and everything else much better here
Lookahead
- dvc style caching for explorer
- maybe separate explorer generation altogether
- overlapping names convention
- resolve naming confusion with colassigner, colaccessor and table feature / composite type / index base classes
- abstract composite type + subclass of entity class
- import ACT, inherit from it and specify
- importing composite type is impossible now if it contains foreign key :(
- add option to infer data type of assigned feature
- can be problematic b/c pandas int/float/nan issue
- create similar sets of features in a dry way
- overlapping in entities
- detect / signal the same type of entity
- exports: postgres, postgis , superset
W3C compliancy plan
- test suite for compliance: https://w3c.github.io/csvw/publishing-snapshots/PR-earl/earl.html
- https://github.com/w3c/csvw
@article{tennison2015model,
title={Model for tabular data and metadata on the web},
author={Tennison, Jeni and Kellogg, Gregg and Herman, Ivan},
year={2015}
}
@article{pollock2015metadata,
title={Metadata vocabulary for tabular data},
author={Pollock, Rufus and Tennison, Jeni and Kellogg, Gregg and Herman, Ivan},
journal={W3C Recommendation},
volume={17},
year={2015}
}
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
datazimmer-0.4.15.tar.gz
(68.4 kB
view hashes)
Built Distribution
Close
Hashes for datazimmer-0.4.15-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0c189c11d4510e8e49ebdac7e2814690d5892ef41ba800f71390861e37f6bab0 |
|
MD5 | 49ae5addd2546e192a05ef45e5185ec9 |
|
BLAKE2b-256 | d6430dd09c2e4d7287ccec5362123dd9b0c12eb69ea9d5024b2d680979192bb4 |