Handy tools for common data engineering needs.
Project description
plumbingbird
Handy tools for common use cases in data engineering
Purpose
I got tired of reinventing wheels across jobs in data engineering, so I decided to make a repo for them instead. Nothing in here is specific business logic, it's intended to be mostly higher order functions so you can plug and play.
Installation Dependencies
[postgresql](https://www.postgresql.org/download/)
[poetry](https://python-poetry.org/docs/#installation)
Organization
Utilities
This directory contains primitive parent classes for concepts in both orchestration and etl, as well as a number of handy tools for environment interaction (like ID-ing where something is running and getting secrets out of the env vars). Someday, maybe the etl and orchestration classes will move to their respective directories, but for now they're just chilling in the base utilities directory.
ETL
This directory contains classes for extraction, transformation, and loading, differentiated by the nature of the source (in the case of extraction), the nature of the destination (in the case of loaders), and the format of the interstitial data (for transformers/buffers).
Orchestration
This directory has provider-bounded tools for standing up cloud services, like Workers that can listen to queues and Jobs they can do.
Tests
What's on the tin, plus some differentiation therein between live tests that take action vs unittests.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for plumbingbird-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c5668b30b6bb7f6c47913094a776e63f436005d8e74d57718e1cfddab9df32d5 |
|
MD5 | c926f7e1e44c3b28eff9aa33dc2c1b4e |
|
BLAKE2b-256 | 24a9ea1821a15340046b012feecc5d9bb8f32cfb5fda07ff7e9775a81dc41367 |