Skip to main content

A tiny mlops library for building machine learning pipelines on your local machine.

Project description

tinypipeline

Overview

tinypipeline is a tiny mlops library that provides a simple framework for organizing your machine learning pipeline code into a series of steps. It does not handle networking, I/O, or compute resources. You do the rest in your pipeline steps.

Installation

$ pip install tinypipeline

Usage

tinypipeline exposes two main objects:

  • pipeline: a decorator for defining your pipeline. Returns a Pipeline instance.
  • step: a function that is used to define individual pipeline steps. Returns a Step instance.

Each object requires you provide a name, version, and description to explicitly define what pipeline you're creating.

The Pipeline object that is returned from the decorator has a single method: run().

API

If you'd like to use this package, you can follow the example.py below:

from tinypipeline import pipeline, step


def step_fn_one():
    print("Step function one")

def step_fn_two():
    print("Step function two")

@pipeline(
    name='test-pipeline', 
    version='0.0.1', 
    description='a test tinypipeline',
)
def pipe():
    step_one = step(
        callable=step_fn_one, 
        name='step_one', 
        version='0.0.1', 
        description='first step',
    )
    step_two = step(
        name='step_two',
        version='0.0.1',
        description='second step',
        callable=step_fn_two,
    )
    return [step_one, step_two]

pipe = pipe()
pipe.run()

Output:

You can run the example.py like so:

$ python example.py
+-------------------------------------------------------------------+
| Running pipeline: Pipeline(name='test-pipeline', version='0.0.1') |
+-------------------------------------------------------------------+

Running step [step_one]...
Step function one
Step [step_one] completed in 0.000356 seconds

Running step [step_two]...
Step function two
Step [step_two] completed in 0.000317 seconds

Running tests

Tests are run using pytest. To run the tests you can do:

$ pip install pytest
$ pytest

Limitations

  • Currently tinypipeline only supports linear pipelines, and doesn't support full dependency graphs

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tinypipeline-0.1.0.tar.gz (6.6 kB view hashes)

Uploaded Source

Built Distribution

tinypipeline-0.1.0-py3-none-any.whl (4.9 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page