Pramen-py
Cli application for defining the data transformations for Pramen.
See:
pramen-py --help
for more information.
Installation
App settings
Application configuration solved by the environment variables (see .env.example)
Add pramen-py as a dependency to your project
In case of poetry:
# ensure we have valid poetry environment
ls pyproject.toml || poetry init
poetry add pramen-py
In case of pip:
pip install pramen-py
Usage
Application configuration
In order to configure the pramen-py options you need to set corresponding environment variables. To see the list of available options run:
pramen-py list-configuration-options
Developing transformations
pramen-py uses python's namespace packages for discovery of the transformations.
This mean, that in order to build a new transformer, it should be located
inside a python package with the transformations directory inside.
This directory should be declared as a package:
- for poetry
[tool.poetry]
# ...
packages = [
{ include = "transformations" },
]
- for setup.py
from setuptools import setup, find_namespace_packages
setup(
name='mynamespace-subpackage-a',
# ...
packages=find_namespace_packages(include=['transformations.*'])
)
Example files structure:
❯ tree .
.
├── README.md
├── poetry.lock
├── pyproject.toml
├── tests
│ └── test_identity_transformer.py
└── transformations
└── identity_transformer
├── __init__.py
└── example_config.yaml
In order to make transformer picked up by the pramen-py the following conditions should be satisfied:
- python package containing the transformers should be installed to the same python environment as pramen-py
- python package should have defined namespace package
transformations - transformers should extend
pramen_py.Transformationbase class
Subclasses created by extending Transformation base class are registered as a cli command (pramen-py transformations run TransformationSubclassName) with default options. Check:
pramen-py transformations run ExampleTransformation1 --help
for more details.
You can add your own cli options to your transformations. See example at ExampleTransformation2
pramen-py pytest plugin
pramen-py also provides pytest plugin with helpful fixtures to test created transformers.
List of available fixtures:
#install pramen-py into the environment and activate it
pytest --fixtures
# check under --- fixtures defined from pramen_py.test_utils.fixtures ---
pramen-py pytest plugin also loads environment variables from .env file if it is presented in the root of the repo.
Running and configuring transformations
Transformations can be run with the following command:
pramen-py transformations run \
ExampleTransformation1 \
--config config.yml \
--info-date 2022-04-01
--config is required option for any transformation. See
config_example.yaml for more information.
To check available options and documentation for a particular transformation, run:
pramen-py transformations run TransformationClassName --help
where TransformationClassName is the name of the transformation.
Using as a Library
Read metastore tables by Pramen-Py API
import datetime
from pyspark.sql import SparkSession
from pramen_py import MetastoreReader
from pramen_py.utils.file_system import FileSystemUtils
spark = SparkSession.getOrCreate()
hocon_config = FileSystemUtils(spark) \
.load_hocon_config_from_hadoop("uri_or_path_to_file")
metastore = MetastoreReader(spark) \
.from_config(hocon_config)
df_txn = metastore.get_table(
"transactions",
info_date_from=datetime.date(2022, 1, 1),
info_date_to=datetime.date(2022, 6, 1)
)
df_customer = metastore.get_latest("customer")
df_txn.show(truncate=False)
df_customer.show(truncate=False)
Development
Prerequisites:
- https://python-poetry.org/docs/#installation
- python 3.6
Setup steps:
git clone https://github.com/AbsaOSS/pramen
cd pramen-py
make install # create virtualenv and install dependencies
make test
make pre-commit
# enable completions
# source <(pramen-py completions zsh)
# source <(pramen-py completions bash)
pramen-py --help
Load environment configuration
Before doing any development step, you have to set your development environment variables
make install
Completions
# enable completions
source <(pramen-py completions zsh)
# or for bash
# source <(pramen-py completions bash)
Deployment
From the local development environment
# bump the version
vim pyproject.toml
# deploy to the dev environment (included steps of building and publishing
# artefacts)
cat .env.ci
make publish
3 maintainers
Metadata
3 maintainers
Release files for pramen-py 1.11.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| pramen_py-1.11.2.tar.gz | 26.8 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| pramen_py-1.11.2-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 72.5 kB
Release files / pramen_py-1.11.2.tar.gz
| Download URL | pramen_py-1.11.2.tar.gz |
|---|---|
| Size | 26.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
342c274c9fc776cd13ccc6db1e1a045c06c8d2b2bc234dfda60b1e33bc2fe5ec
|
|
BLAKE2b-256 checksum How to use checksums |
ca23791b39c38c6dba86460bb51c8a401df1d4d880563f91a89f3d88a4dd031c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.4.2 CPython/3.10.16 Linux/6.8.0-1021-azure
|
Release files / pramen_py-1.11.2-py3-none-any.whl
| Download URL | pramen_py-1.11.2-py3-none-any.whl |
|---|---|
| Size | 45.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
710949509aa97d7bf4fb5c01d0d5e2f12ccbdca9ada5560ba5f43dc44fcaf431
|
|
BLAKE2b-256 checksum How to use checksums |
357cbd78958ea6dbf7a07dbe7195e9c64910e6a9f9636809587fe16fccafabc3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.4.2 CPython/3.10.16 Linux/6.8.0-1021-azure
|
Security for pramen-py 1.11.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
