Project Status: Archived
This repository is no longer actively maintained. Due to shifting priorities and limited resources, we have decided to archive the repository and discontinue further development and maintenance.
What this means:
- No new features or updates will be added.
- Issues and pull requests will no longer be reviewed or responded to.
- You are welcome to fork the project and continue development under your own maintenance.
luisy
This tool is an extension for the Python Framework luigi which helps to build reproducable and complex data pipelines for batch jobs. Visit our docs to learn more!
How to use?
This is how an end-to-end luisy pipeline may look like:
import luisy
import pandas as pd
@luisy.raw
@luisy.csv_output(delimiter=',')
class InputFile(luisy.ExternalTask):
label = luisy.Parameter()
def get_file_name(self):
return f"file_{self.label}"
@luisy.interim
@luisy.requires(InputFile)
class ProcessedFile(luisy.Task):
def run(self):
df = self.input().read()
# Some more preprocessings
# ...
# Write to disk
self.write(df)
@luisy.final
class MergedFile(luisy.ConcatenationTask):
def requires(self):
for label in ['a', 'b', 'c', 'd']:
yield ProcessedFile(label=label)
How to install?
Stable Branch: main
Minimum python version: 3.8
Install luisy with
pip install luisy
How to test?
To run all unittests that are inside the tests directory use the following command:
pytest
How to contribute?
Please have a look at our contribution guide.
Third-Party Licenses
Runtime dependencies
| Name | License | Type |
|---|---|---|
| numpy | BSD-3-Clause License | Dependency |
| pandas | BSD 3-Clause License | Dependency |
| networkx | BSD-3-Clause License | Dependency |
| luigi | Apache License 2.0 | Dependency |
| distlib | Python license | Dependency |
| matplotlib | Other | Dependency |
| azure-storage-blob | MIT License | Dependency |
| tables | BSD license | Dependency |
| pipdeptree | MIT License | Dependency |
| requirements-parser | Apache License 2.0 | Dependency |
| pyarrow | Apache License 2.0 | Dependency |
| spark | Apache License 2.0 | Dependency |
Development dependency
| Name | License | Type |
|---|---|---|
| sphinx | BSD-2-Clause | Dependency |
| sphinx_rtd_theme | MIT License | Dependency |
| flake8 | MIT License | Dependency |
| pytest | MIT License | Dependency |
| pytest-flake8 | BSD License | Dependency |
| pytest-cov | MIT License | Dependency |
| pip-tools | BSD 3-Clause License | Dependency |
1 maintainer
Metadata
1 maintainer
Release files for luisy 1.4.7
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| luisy-1.4.7.tar.gz | 41.1 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| luisy-1.4.7-py2.py3-none-any.whl | Python 2, Python 3 | none | any | Details |
Total release size: 88.2 kB
Release files / luisy-1.4.7.tar.gz
| Download URL | luisy-1.4.7.tar.gz |
|---|---|
| Size | 41.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
b1209b4749cbfd1f48304b753b6484b5e8c1fd871eb225b4e8fb8b228280c18a
|
|
BLAKE2b-256 checksum How to use checksums |
fb9600961d2f6e9d3a55f9ca5609cbdbee777401d87bdb4d04391e44a997d19d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
Release files / luisy-1.4.7-py2.py3-none-any.whl
| Download URL | luisy-1.4.7-py2.py3-none-any.whl |
|---|---|
| Size | 47.1 kB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
edea81f7d50855e85270d4616acf15554657fb168cb64917598412783ff2c4b3
|
|
BLAKE2b-256 checksum How to use checksums |
8442a9baf8638adbed103a37062be11e0a10696a6f73b9b18e21254f01911a45
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
