datazimmer 0.4.12

sscu-budapest utilities for scientific data engineering

datazimmer

To create a new project

• make sure that python points to python>=3.8 and you have pip and git then pip install datazimmer
• run dz init project-name
  • pulls project-template
• add a remote
  • both to git and dvc (can run dz build-meta to see available dvc remotes)
  • git remote can be given with dz init
• create, register and document steps in a pipeline you will run in different environments
• build metadata to exportable and serialized format with dz build-meta
  • if you defined importable data from other artifacts in the config, you can import them with load-external-data
  • ensure that you import envs that are served from sources you have access to
• build and run pipeline steps by running dz run
• validate that the data matches the datascript description with dz validate

Scheduling

• a project as a whole has a cron expression in zimmer.yaml to determine the schedule of reruns
• additionally, aswan projects within the dz project can have different cron expressions for scheduling new runs of the aswan projects

Test projects

TODO: document dogshow and everything else much better here

Lookahead

• dvc style caching for explorer
  • maybe separate explorer generation altogether
• overlapping names convention
• resolve naming confusion with colassigner, colaccessor and table feature / composite type / index base classes
• abstract composite type + subclass of entity class
  • import ACT, inherit from it and specify
  • importing composite type is impossible now if it contains foreign key :(
• add option to infer data type of assigned feature
  • can be problematic b/c pandas int/float/nan issue
• create similar sets of features in a dry way
• overlapping in entities
  • detect / signal the same type of entity
• exports: postgres, postgis , superset

W3C compliancy plan

• test suite for compliance: https://w3c.github.io/csvw/publishing-snapshots/PR-earl/earl.html
• https://github.com/w3c/csvw
  • https://www.w3.org/TR/2015/REC-tabular-data-model-20151217/
  • https://www.w3.org/TR/tabular-metadata/

@article{tennison2015model,
  title={Model for tabular data and metadata on the web},
  author={Tennison, Jeni and Kellogg, Gregg and Herman, Ivan},
  year={2015}
}

@article{pollock2015metadata,
  title={Metadata vocabulary for tabular data},
  author={Pollock, Rufus and Tennison, Jeni and Kellogg, Gregg and Herman, Ivan},
  journal={W3C Recommendation},
  volume={17},
  year={2015}
}