tiny-blocks
Tiny Blocks to build large and complex ETL data pipelines!
Tiny-Blocks is a library for data engineering operations.
Each pipeline is made out of tiny-blocks glued with the >> operator.
This library relies on a fundamental streaming abstraction consisting of three
parts: extract, transform, and load. You can view a pipeline
as an extraction, followed by zero or more transformations, followed by a sink.
Visually, this looks like:
extract -> transform1 -> transform2 -> ... -> transformN -> load
You can also fan-in, fan-out for more complex operations.
extract1 -> transform1 -> |-> transform2 -> ... -> | -> transformN -> load1
extract2 ---------------> | | -> load2
Tiny-Blocks use generators to stream data. Each chunk is a Pandas DataFrame.
The chunksize or buffer size is adjustable per pipeline.
Installation
Install it using pip
pip install tiny-blocks
Basic usage
from tiny_blocks.extract import FromCSV
from tiny_blocks.transform import Fillna
from tiny_blocks.load import ToSQL
# ETL Blocks
from_csv = FromCSV(path='/path/to/source.csv')
fill_na = Fillna(value="Hola Mundo")
to_sql = ToSQL(dsn_conn='psycopg2+postgres://...', table_name="sink")
# Pipeline
from_csv >> fill_na >> to_sql
Examples
For more complex examples please visit the notebooks' folder.
Documentation
Please visit this link for documentation.
1 maintainer
Metadata
1 maintainer
Release files for tiny-blocks 0.1.15
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| tiny_blocks-0.1.15.tar.gz | 13.9 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| tiny_blocks-0.1.15-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 38.6 kB
Release files / tiny_blocks-0.1.15.tar.gz
| Download URL | tiny_blocks-0.1.15.tar.gz |
|---|---|
| Size | 13.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
963b99a1ea84e55a56aec99f10de102efd66756554e3831eb680142b7544d82e
|
|
BLAKE2b-256 checksum How to use checksums |
9a469e401478d8636c9ea0f96c6ff22b1313eda591dcb8b4ab8b7b6ce0bd314e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.1.14 CPython/3.9.10 Darwin/21.6.0
|
Release files / tiny_blocks-0.1.15-py3-none-any.whl
| Download URL | tiny_blocks-0.1.15-py3-none-any.whl |
|---|---|
| Size | 24.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
59322ee078cb053939e768e80adf599cc401bdab6c6c37012cf73c8c56a389f9
|
|
BLAKE2b-256 checksum How to use checksums |
93bfce32b3ca2b8faeeae5deadc51db281f6e15ef2de708d98ddd81719f32151
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.1.14 CPython/3.9.10 Darwin/21.6.0
|
