Skip to main content

dalymi

[data like you mean it]

Documentation Status

A lightweight, data-focused and non-opinionated pipeline manager written in and for Python.


dalymi allows to build data processing pipelines as directed acyclic graphs (DAGs) and facilitates rapid, but controlled, model development. The goal is to prototype quickly, but scale to production with ease. To achieve this, dalymi uses "make"-style workflows, i.e. tasks with missing input trigger the execution of input-producing tasks before being executed themselves. At the same time, dalymi provides fine control to run and undo specific pipeline parts for quick test iterations. This ensures output reproducability and minimizes manual errors.

Several features facilitate dalymi's goal:

  • simple, non-opinionated API (most choices left to user)
  • no external dependencies for pipeline execution
  • one-line installation (ready for use)
  • no configuration
  • auto-generated command line interface for pipeline execution
  • quick start, but high flexibility to customize and extend:
    • task output can be stored in any format Python can touch (local files being the default)
    • customizable command line arguments
    • templated output location (e.g. timestamped files)
    • support for automated checks on data integrity during runtime
  • DAG visualization using graphviz
  • API design encourages good development practices (modular code, defined data schemas, self-documenting code, easy workflow viz, etc.)

Installation

dalymi requires Python >= 3.5.

pip install dalymi

For the latest development:

pip install git+https://github.com/joschnitzbauer/dalymi.git

Documentation

http://dalymi.readthedocs.io/

Simple example

simple.py:

from dalymi import Pipeline
from dalymi.resources import PandasCSV
import pandas as pd


# Define resources:
numbers_resource = PandasCSV(name='numbers', loc='numbers.csv', columns=['number'])
squares_resource = PandasCSV(name='squares', loc='squares.csv', columns=['number', 'square'])


# Define the pipeline
pl = Pipeline()


@pl.output(numbers_resource)
def create_numbers(**context):
    return pd.DataFrame({'number': range(11)})


@pl.output(squares_resource)
@pl.input(numbers_resource)
def square_numbers(numbers, **context):
    numbers['square'] = numbers['number']**2
    return numbers


if __name__ == '__main__':
    # Run the default command line interface
    pl.cli()

Command line:

python simple.py run     # executes the pipeline. skips tasks for which output already exists.

More examples can be found here.

Roadmap

  • More docstrings
  • Unit tests
  • Continuous integration
  • Parallel task processing
  • REST API during pipeline run
  • Web interface for pipeline run

Warranty

Although dalymi is successfully used in smaller applications, it is not battle-tested yet and lacks unit tests. If you decide to use it, be prepared to communicate issues or fix bugs (it's not a lot of code... :)).

Contributions

... are welcome!

Metadata

Release files for dalymi 0.1.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for dalymi 0.1.5
File Size Uploaded Provenance Actions
dalymi-0.1.5.tar.gz 6.3 kB Without provenance Details for dalymi-0.1.5.tar.gz

Release files / dalymi-0.1.5.tar.gz

Download URL dalymi-0.1.5.tar.gz
Size 6.3 kB
Tags Source
SHA-256 checksum
How to use checksums
7ae37fdb54b2c27e2b40c4831cc5b0d1ac030d4711a3ed38facc86ab64f4293f
BLAKE2b-256 checksum
How to use checksums
2df07869f5a28067a4f45fbfec79a0bac43a3dde35cdef49052e38a50ebbe6c0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/40.2.0 requests-toolbelt/0.9.1 tqdm/4.38.0 CPython/3.5.6

Security for dalymi 0.1.5

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page