Pipelines for machine learning workloads.
Project description
Pipeline
Table of Contents
About
Pipeline is a python library that provides a simple way to construct computational graphs for AI/ML. The library is suitable for both development and production environments supporting inference and training/finetuning. This library is also a direct interface to Pipeline.ai which provides a compute engine to run pipelines at scale and on enterprise GPUs.
The syntax used for defining AI/ML pipelines shares some similarities in syntax to sessions in Tensorflow v1, and Flows found in Prefect. In future releases we will be moving away from this syntax to a C based graph compiler which interprets python directly (and other languages) allowing users of the API to compose graphs in a more native way to the chosen language.
Quickstart
:warning: Uploading pipelines to Pipeline Cloud works best in Python 3.9. We strongly recommend you use Python 3.9 when uploading pipelines because the
pipeline-ai
library is still in beta and is known to cause opaque errors when pipelines are serialised from a non-3.9 environment.
Basic maths
from pipeline import Pipeline, Variable, pipeline_function
@pipeline_function
def square(a: float) -> float:
return a**2
@pipeline_function
def multiply(a: float, b: float) -> float:
return a * b
with Pipeline("maths") as pipeline:
flt_1 = Variable(type_class=float, is_input=True)
flt_2 = Variable(type_class=float, is_input=True)
pipeline.add_variables(flt_1, flt_2)
sq_1 = square(flt_1)
res_1 = multiply(flt_2, sq_1)
pipeline.output(res_!)
output_pipeline = Pipeline.get_pipeline("maths")
print(output_pipeline.run(5.0, 6.0))
Transformers (GPT-Neo 125M)
Note: requires torch
and transformers
as dependencies.
from pipeline import Pipeline, Variable
from pipeline.objects.huggingface.TransformersModelForCausalLM import (
TransformersModelForCausalLM,
)
with Pipeline("HF pipeline") as builder:
input_str = Variable(str, is_input=True)
model_kwargs = Variable(dict, is_input=True)
builder.add_variables(input_str, model_kwargs)
hf_model = TransformersModelForCausalLM(
model_path="EleutherAI/gpt-neo-125M",
tokenizer_path="EleutherAI/gpt-neo-125M",
)
hf_model.load()
output_str = hf_model.predict(input_str, model_kwargs)
builder.output(output_str)
output_pipeline = Pipeline.get_pipeline("HF pipeline")
print(
output_pipeline.run(
"Hello my name is", {"min_length": 100, "max_length": 150, "temperature": 0.5}
)
)
Installation instructions
Linux, Mac (intel)
pip install -U pipeline-ai
Mac (arm/M1)
Due to the ARM architecture of the M1 core it is necessary to take additional steps to install Pipeline, mostly due to the transformers library. We recoomend running inside of a conda environment as shown below.
- Make sure Rosetta2 is disabled.
- From terminal run:
xcode-select --install
- Install Miniforge, instructions here: https://github.com/conda-forge/miniforge or follow the below:
- Download the Miniforge install script here: https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh
- Make the shell executable and run
sudo chmod 775 Miniforge3-MacOSX-arm64.sh ./Miniforge3-MacOSX-arm64.sh
- Create a conda based virtual env and activate:
conda create --name pipeline-env python=3.9
conda activate pipeline-env
- Install tensorflow
conda install -c apple tensorflow-deps
python -m pip install -U pip
python -m pip install -U tensorflow-macos
python -m pip install -U tensorflow-metal
- Install transformers
conda install -c huggingface transformers -y
- Install pipeline
python -m pip install -U pipeline-ai
Features
Future roadmap
Currently working on:
- Custom environments
- Reduced cold start in pytorch via custom tensor loading
- CLI authentication
Working on next:
- Remote reference of objects in the
Pipeline
context manager - Remote events
- Finetuning premade pipelines
- Conditional logic flow in
Pipeline
context manager - Pipeline chaining (call another pipeline in a pipeline)
More ideas:
- Deployment control via CLI
- Add in a
PipelineDirectory
object - Streaming pipeline runs
- Logs on remote runs
- Model deployment from CLI
pipeline deploy FILE.py
Development
This project is made with poetry, so firstly setup poetry on your machine.
Once that is done, please run
sh setup.sh
With this you should be good to go. This sets up dependencies, pre-commit hooks and pre-push hooks.
You can manually run pre commit hooks with
pre-commit run --all-files
To run tests manually please run
pytest
License
Pipeline is licensed under Apache Software License Version 2.0.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for pipeline_ai-0.2.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4b3a8c0038d2d61a8440f11ca889a63942db2081909e7c8bcde3be33ae03635e |
|
MD5 | 270a3beb073ce6f3d7f4672445ffbf02 |
|
BLAKE2b-256 | a8c751f3edf3e43e59ea5f43740f83ff893e7c514a283381e940e9e432908f01 |