Skip to main content

sscu-budapest utilities for scientific data engineering

Project description

datazimmer

Documentation Status codeclimate codecov pypi DOI

To create a new project

  • make sure that python points to python>=3.8 and you have pip and git then pip install datazimmer
  • run dz init project-name
  • add a remote
    • both to git and dvc (can run dz build-meta to see available dvc remotes)
    • git remote can be given with dz init
  • create, register and document steps in a pipeline you will run in different environments
  • build metadata to exportable and serialized format with dz build-meta
    • if you defined importable data from other artifacts in the config, you can import them with load-external-data
    • ensure that you import envs that are served from sources you have access to
  • build and run pipeline steps by running dz run
  • validate that the data matches the datascript description with dz validate

Scheduling

  • a project as a whole has a cron expression in zimmer.yaml to determine the schedule of reruns
  • additionally, aswan projects within the dz project can have different cron expressions for scheduling new runs of the aswan projects

Test projects

TODO: document dogshow and everything else much better here

Lookahead

  • dvc style caching for explorer
    • maybe separate explorer generation altogether
  • overlapping names convention
  • resolve naming confusion with colassigner, colaccessor and table feature / composite type / index base classes
  • abstract composite type + subclass of entity class
    • import ACT, inherit from it and specify
    • importing composite type is impossible now if it contains foreign key :(
  • add option to infer data type of assigned feature
    • can be problematic b/c pandas int/float/nan issue
  • create similar sets of features in a dry way
  • overlapping in entities
    • detect / signal the same type of entity
  • exports: postgres, postgis , superset

W3C compliancy plan

@article{tennison2015model,
  title={Model for tabular data and metadata on the web},
  author={Tennison, Jeni and Kellogg, Gregg and Herman, Ivan},
  year={2015}
}
@article{pollock2015metadata,
  title={Metadata vocabulary for tabular data},
  author={Pollock, Rufus and Tennison, Jeni and Kellogg, Gregg and Herman, Ivan},
  journal={W3C Recommendation},
  volume={17},
  year={2015}
}

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

datazimmer-0.4.12.tar.gz (67.9 kB view hashes)

Uploaded Source

Built Distribution

datazimmer-0.4.12-py3-none-any.whl (55.6 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page