Facilitate data engineering on the Ingenii Data Platform
Project description
Ingenii Data Engineering Package
Details
- Current Version: 0.3.2
Overview
This package provides utilities for data engineering on Ingenii's Azure Data Platform. This can be both used for local development, and is used in the Ingenii Databricks Runtime.
Usage
Import the package to use the functions within.
import ingenii_data_engineering
dbt
Part of this package validates dbt schemas to ensure they are compatible with Databricks and the larger Ingenii Data Platform. This happens when a data pipeline to ingest a file is run, to make sure a file is ingested correctly. Full details of how to set up your dbt schema files in your Data Engineering repository can be found in the Ingenii Data Engineering Example repository.
Pre-processing
This package contains code to facilitate the pre-processing of files before they are ingested by the data platform. This allows users to transform any data into a form that is compatible. See details of working with pre-processing functions in the Ingenii Data Engineering Example repository.
This package also contains the code to turn the pre-processing scripts into a package, ready to be uploaded and used by the Data Platform. Once this package is installed, the command
python -m <package name> <command> <folder with pre-processing code>
python -m ingenii_data_engineering pre_processing_package pre_process
will generate a .whl
file in a folder called dist/
. For more details, see the Ingenii Data Engineering Example repository.
Development
Prerequisites
- A working knowledge of git SCM
- Installation of Python 3.7.3
Set up
- Complete the 'Getting Started > Prerequisites' section
- For Windows only:
- Run
make setup
: to copy the .env into place (.env-dist
>.env
)
Getting started
-
Complete the 'Getting Started > Set up' section
-
From the root of the repository, in a terminal (preferably in your IDE) run the following commands to set up a virtual environment:
python -m venv venv . venv/bin/activate pip install -r requirements-dev.txt pre-commit install
or for Windows:
python -m venv venv . venv/Scripts/activate pip install -r requirements-dev.txt pre-commit install
-
Note: if you get a
permission denied
error when executing thepre-commit install
command you'll need to runchmod -R 775 venv/bin/
to recursively update permissions in thevenv/bin/
dir -
The following checks are run as part of pre-commit hooks: flake8(note unit tests are not run as a hook)
Building
- Complete the 'Getting Started > Set up' section
- Run
make build
to create the package in./dist
- Run
make clean
to remove dist files
Testing
- Complete the 'Getting Started > Set up' and 'Development' sections
- Run
make test
to run the unit tests using pytest - Run
flake8
to run lint checks using flake8 - Run
make qa
to run the unit tests and linting in a single command - Run
make qa
to remove pytest files
Version History
0.3.2
: Further bugfix for JSON UTF-8 BOM0.3.1
: Remove unnecessary functions specific to Databricks0.3.0
: Create pre-processing package using the module0.2.1
: Handle JSON read UTF-8 BOM0.2.0
: Pre-processing happens all in the 'archive' container0.1.5
: Better functionality for column names in .csv files0.1.4
: Handle JSON files0.1.3
: Adding pre-processing utilities0.1.2
: Rearrangement and better split of work with the Databricks Runtime. Better validation0.1.1
: Minor bug fixes0.1.0
: dbt schema validation, pre-processing class
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for ingenii_data_engineering-0.3.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5f5179b566aca794cf4f8d00ace788305f1efa9b1008eb70a6f1245933a547ea |
|
MD5 | e780ca4fb7dc5865f678069bb135d9d8 |
|
BLAKE2b-256 | 2e365ecf72894888f13cfd5f5ad756b6516cca8cb6b4df8662bc7d7c80b42507 |
Hashes for ingenii_data_engineering-0.3.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0f0e0dd9839295461eb08558281c03500fc958e20e751f9739b87e34b128860d |
|
MD5 | 0f329118e860ce36e81dd19722d22a8d |
|
BLAKE2b-256 | fe719397935d170ce33dc5527160e4c7d3b00d4d1850011e78f82eafab750593 |