dalymi
[data like you mean it]
A lightweight, data-focused and non-opinionated pipeline manager written in and for Python.
dalymi allows to build data processing pipelines as directed acyclic graphs (DAGs) and facilitates rapid, but controlled, model development. The goal is to prototype quickly, but scale to production with ease. To achieve this, dalymi uses "make"-style workflows, i.e. tasks with missing input trigger the execution of input-producing tasks before being executed themselves. At the same time, dalymi provides fine control to run and undo specific pipeline parts for quick test iterations. This ensures output reproducability and minimizes manual errors.
Several features facilitate dalymi's goal:
- simple, non-opinionated API (most choices left to user)
- no external dependencies for pipeline execution
- one-line installation (ready for use)
- no configuration
- auto-generated command line interface for pipeline execution
- quick start, but high flexibility to customize and extend:
- task output can be stored in any format Python can touch (local files being the default)
- customizable command line arguments
- templated output location (e.g. timestamped files)
- support for automated checks on data integrity during runtime
- DAG visualization using graphviz
- API design encourages good development practices (modular code, defined data schemas, self-documenting code, easy workflow viz, etc.)
Installation
dalymi requires Python >= 3.5.
pip install dalymi
For the latest development:
pip install git+https://github.com/joschnitzbauer/dalymi.git
Documentation
Simple example
simple.py:
from dalymi import Pipeline
from dalymi.resources import PandasCSV
import pandas as pd
# Define resources:
numbers_resource = PandasCSV(name='numbers', loc='numbers.csv', columns=['number'])
squares_resource = PandasCSV(name='squares', loc='squares.csv', columns=['number', 'square'])
# Define the pipeline
pl = Pipeline()
@pl.output(numbers_resource)
def create_numbers(**context):
return pd.DataFrame({'number': range(11)})
@pl.output(squares_resource)
@pl.input(numbers_resource)
def square_numbers(numbers, **context):
numbers['square'] = numbers['number']**2
return numbers
if __name__ == '__main__':
# Run the default command line interface
pl.cli()
Command line:
python simple.py run # executes the pipeline. skips tasks for which output already exists.
More examples can be found here.
Roadmap
- More docstrings
- Unit tests
- Continuous integration
- Parallel task processing
- REST API during pipeline run
- Web interface for pipeline run
Warranty
Although dalymi is successfully used in smaller applications, it is not battle-tested yet and lacks unit tests. If you decide to use it, be prepared to communicate issues or fix bugs (it's not a lot of code... :)).
Contributions
... are welcome!
Metadata
Release files for dalymi 0.1.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| dalymi-0.1.5.tar.gz | 6.3 kB | Without provenance | Details for dalymi-0.1.5.tar.gz |
Release files / dalymi-0.1.5.tar.gz
| Download URL | dalymi-0.1.5.tar.gz |
|---|---|
| Size | 6.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7ae37fdb54b2c27e2b40c4831cc5b0d1ac030d4711a3ed38facc86ab64f4293f
|
|
BLAKE2b-256 checksum How to use checksums |
2df07869f5a28067a4f45fbfec79a0bac43a3dde35cdef49052e38a50ebbe6c0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/40.2.0 requests-toolbelt/0.9.1 tqdm/4.38.0 CPython/3.5.6
|
Security for dalymi 0.1.5
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.