analytics-mesh 0.1.12

Facades and common functions necessary for data science and data engineering workflows

analytics-mesh

Interfaces and facades that facilitate a common approach to analytics tasks

Getting Started

Please install the requirements:

pip install -r requirements

If you are going to be making modifications to the ipynb notebooks, then be sure to install the pre-commit hook (see below):

pre-commit install

Tests

Tests are currently split into unit and integration tests. As this package integrates with storage systems, the integration tests are typically running against things like Google Cloud Platform.

Tests may be run in the root folder of the repo with:

coverage run -m unittest discover tests && coverage report

If you want to run just the unit tests (and ignore coverage) then,

python -m unittest discover tests/unit

Similarly, for the integration tests.

Pre-Commit Hooks and Notebook Workflow

In this repo we are using the python precommit package (included in requirements.txt file). In order to leverage it in your development workflow, you need to run the following commands (assuming you have already installed your requirements).

pre-commit install

We follow the convention that a version controlled ipynb file is converted to a markdown (md) file to form an ipynb-md pair that are both version controlled. This allows us to code review the markdown files whilst keeping the original ipynb file output intact for easy perusal on the repository manager (Gitlab in our case).

Packaging the Code

The contents of the mesh package are packaged in the build pipeline and submitted to pypi when merged to the master branch. The contents of the demos folder and tests is omitted from the package. See the .gitlab-ci.yml file for the pypi instructions we use, as well as the tests and linting that are performed.

The consequences of this is that you will convert ipynb notebooks to md.

Contributions

Contributions are most welcome. Please submit patches or new features for code review by any of the main contributors:

• Jacques du Toit
• Carl du Plessis
• Jaco Gericke

Please be sure to run the tests prior to submission.