Skip to main content

[data like you mean it] A lightweight, data-focused and non-opinionated pipeline manager written in and for Python.

Project description


[data like you mean it]

Documentation Status

A lightweight, data-focused and non-opinionated pipeline manager written in and for Python.

dalymi allows to build data processing pipelines as directed acyclic graphs (DAGs) and facilitates rapid, but controlled, model development. The goal is to prototype quickly, but scale to production with ease. To achieve this, dalymi uses "make"-style workflows, i.e. tasks with missing input trigger the execution of input-producing tasks before being executed themselves. At the same time, dalymi provides fine control to run and undo specific pipeline parts for quick test iterations. This ensures output reproducability and minimizes manual errors.

Several features facilitate dalymi's goal:

  • simple, non-opinionated API (most choices left to user)
  • no external dependencies for pipeline execution
  • one-line installation (ready for use)
  • no configuration
  • auto-generated command line interface for pipeline execution
  • quick start, but high flexibility to customize and extend:
    • task output can be stored in any format Python can touch (local files being the default)
    • customizable command line arguments
    • templated output location (e.g. timestamped files)
    • support for automated checks on data integrity during runtime
  • DAG visualization using graphviz
  • API design encourages good development practices (modular code, defined data schemas, self-documenting code, easy workflow viz, etc.)


dalymi requires Python >= 3.5.

pip install dalymi

For the latest development:

pip install git+


Simple example

from dalymi import Pipeline
from dalymi.resources import PandasCSV
import pandas as pd

# Define resources:
numbers_resource = PandasCSV(name='numbers', loc='numbers.csv', columns=['number'])
squares_resource = PandasCSV(name='squares', loc='squares.csv', columns=['number', 'square'])

# Define the pipeline
pl = Pipeline()

def create_numbers(**context):
    return pd.DataFrame({'number': range(11)})

def square_numbers(numbers, **context):
    numbers['square'] = numbers['number']**2
    return numbers

if __name__ == '__main__':
    # Run the default command line interface

Command line:

python run     # executes the pipeline. skips tasks for which output already exists.

More examples can be found here.


  • More docstrings
  • Unit tests
  • Continuous integration
  • Parallel task processing
  • REST API during pipeline run
  • Web interface for pipeline run


Although dalymi is successfully used in smaller applications, it is not battle-tested yet and lacks unit tests. If you decide to use it, be prepared to communicate issues or fix bugs (it's not a lot of code... :)).


... are welcome!

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for dalymi, version 0.1.5
Filename, size File type Python version Upload date Hashes
Filename, size dalymi-0.1.5.tar.gz (6.3 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page