Yay ml pipelines
Project description
Pipeline
About
Pipeline is a python library that provides a simple way to construct computational graphs for AI/ML. The library is suitable for both development and production environments supporting inference and training/finetuning. This library is also a direct interface to Pipeline.ai which provides a compute engine to run pipelines at scale and on enterprise GPUs.
The syntax used for defining AI/ML pipelines shares some similarities in syntax to sessions in Tensorflow v1, and Flows found in Prefect. In future releases we will be moving away from this syntax to a C based graph compiler which interprets python directly (and other languages) allowing users of the API to compose graphs in a more native way to the chosen language.
Usage
Huggingface Transformers
from pipeline import Pipeline, Variable, pipeline_function, for_loop
from pipeline.model.transformer_models import TransformersModel
with Pipeline(pipeline_name="GPTNeo") as pipeline:
input_str = Variable(variable_type=str, is_input=True)
hf_model = TransformersModel("EleutherAI/gpt-neo-125M", "EleutherAI/gpt-neo-125M")
output_str = hf_model.predict(input_str)
pipeline.output(output_str)
output_pipeline = Pipeline.get_pipeline("GPTNeo")
print(output_pipeline.run("Hello"))
Installation instructions
Linux, Mac (intel)
pip install -U pipeline-ai
Mac (arm/M1)
Due to the ARM architecture of the M1 core it is necessary to take additional steps to install Pipeline, mostly due to the transformers library. We recoomend running inside of a conda environment as shown below.
- Make sure Rosetta2 is disabled.
- From terminal run:
xcode-select --install
- Install Miniforge, instructions here: https://github.com/conda-forge/miniforge or follow the below:
- Download the Miniforge install script here: https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh
- Make the shell executable and run
sudo chmod 775 Miniforge3-MacOSX-arm64.sh ./Miniforge3-MacOSX-arm64.sh
- Create a conda based virtual env and activate:
conda create --name pipeline-env python=3.9
conda activate pipeline-env
- Install tensorflow
conda install -c apple tensorflow-deps
python -m pip install -U pip
python -m pip install -U tensorflow-macos
python -m pip install -U tensorflow-metal
- Install transformers
conda install -c huggingface transformers -y
- Install pipeline
python -m pip install -U pipeline-ai
License
Pipeline is licensed under Apache Software License Version 2.0.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for pipeline_ai-0.0.12-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f984019240527fca61bae2187ea5a194deb6b0d763531dde494d8e4e60426935 |
|
MD5 | 4fb934224b4d9595bc7cab43e1d3c35c |
|
BLAKE2b-256 | 706521b6b5f3433617c87ee2648755d65643d02910bcc7f29f5c614eac985898 |