etl-entities 1.3.1

ETL Entities lib for onETL

What is ETL Entities?

Collection of classes used for handling High Water Mark (HWM) and gathering Lineage graph.

Currently implemented:

• IntHWM

• DateHWM

• DateTimeHWM

• FileListHWM

• Column

• Table

• RemoteFolder

• Process

How to install

pip install etl-entities

Documentation

See https://etl-entities.readthedocs.io/

Contribution guide

See CONTRIBUTING.rst

Security

See SECURITY.rst

Develop

Clone repo

Clone repo:

git clone [e-mail]:MobileTeleSystems/etl-entities.git -b develop

cd etl-entities

Setup environment

Create virtualenv and install dependencies:

# create virtual environment
python -m venv venv
source venv/bin/activate
pip install -U wheel
pip install -U pip setuptools

# install requirements
pip install -U -r requirements.txt

Install dependencies for development:

# install linters, formatters, etc
pip install -U -r requirements-dev.txt

Enable pre-commit hooks

Install pre-commit hooks:

pre-commit install --install-hooks

Test pre-commit hooks run:

pre-commit run

Run tests

# install requirements for testing
pip install -U -r requirements-test.txt

# run tests
pytest

Build documentation

# install requirements for documentation
pip install -U -r requirements-docs.txt

cd docs

# generate html documentation
make html

Then open docs/_build/html/index.html file in browser.