datazimmer 0.4.8

sscu-budapest utilities for scientific data engineering

datazimmer

Some utility function to help with

• setting up data environments
• simplified dvc pipeline registry

these are used in the project-template

Make sure that python points to python>=3.8 and you have pip and git

Functions

Tinker

check out a table or few, with a notebook and some basic analysis to help

Engineer Research

Scheduling

• a project as a whole has a cron expression to determine the schedule of reruns
• additionally, aswan projects within the dz project can have different cron expressions for scheduling new runs of the aswan projects

Lookahead

• overlapping names convention
• resolve naming confusion with colassigner, colaccessor and table feature / composite type / index base classes
• abstract composite type + subclass of entity class
  • import ACT, inherit from it and specify
  • importing composite type is impossible now if it contains foreign key :(
• automatic filter for env creation based on foreign key metadata
• add option to infer data type of assigned feature
  • can be problematic b/c pandas int/float/nan issue
• sharing functions among projects
  • functions specific to processing certain composite / named types
  • e.g. function dealing with fitting into a limit in dogshow project 1
• create similar sets of features in a dry way
• detecting reliance of composite type given by assigner
  • can wait, as initial import is just the assigner transformed to accessor
• overlapping in entities
  • detect / signal the same type of entity
• properly assert importing