A tiny mlops library for building machine learning pipelines on your local machine.
Project description
tinypipeline
Overview
tinypipeline
is a tiny mlops library that provides a simple framework for organizing your machine learning pipeline code into a series of steps. It does not handle networking, I/O, or compute resources. You do the rest in your pipeline steps.
Installation
$ pip install tinypipeline
Usage
tinypipeline
exposes two main objects:
pipeline
: a decorator for defining your pipeline. Returns aPipeline
instance.step
: a function that is used to define individual pipeline steps. Returns aStep
instance.
Each object requires you provide a name
, version
, and description
to explicitly define what pipeline you're creating.
The Pipeline
object that is returned from the decorator has a single method: run()
.
API
If you'd like to use this package, you can follow the example.py
below:
from tinypipeline import pipeline, step
def step_fn_one():
print("Step function one")
def step_fn_two():
print("Step function two")
@pipeline(
name='test-pipeline',
version='0.0.1',
description='a test tinypipeline',
)
def pipe():
step_one = step(
callable=step_fn_one,
name='step_one',
version='0.0.1',
description='first step',
)
step_two = step(
name='step_two',
version='0.0.1',
description='second step',
callable=step_fn_two,
)
return [step_one, step_two]
pipe = pipe()
pipe.run()
Output:
You can run the example.py
like so:
$ python example.py
+-------------------------------------------------------------------+
| Running pipeline: Pipeline(name='test-pipeline', version='0.0.1') |
+-------------------------------------------------------------------+
Running step [step_one]...
Step function one
Step [step_one] completed in 0.000356 seconds
Running step [step_two]...
Step function two
Step [step_two] completed in 0.000317 seconds
Running tests
Tests are run using pytest. To run the tests you can do:
$ pip install pytest
$ pytest
Limitations
- Currently
tinypipeline
only supports linear pipelines, and doesn't support full dependency graphs
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for tinypipeline-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fcf4a5f12a0c40a567b32ac272f632338065b44c7939f76c30c5a4563a0eca01 |
|
MD5 | ae7b77ee6f77a2344fd5ae543c73190a |
|
BLAKE2b-256 | f21c28f91d034dda6aeab88a293e85a4a45f40d024621725b41c6c92cbb7ccb4 |