A simple pipeline execution library
Project description
pipeshell
pipeshell is a simple pipeline execution library allowing you to define and run pipelines of shell commands with dependencies, retries, and other features.
Features
- Define steps with shell commands
- Set dependencies between steps
- Retry steps on failure
- Run steps in the background
- Capture and display step outputs
- Set environment variables for steps
Installation
To install pipeshell
, you can use pip
:
pip install pipeshell
Usage
Here's a quick example of how to define and run a pipeline with pipeshell
:
from pipeshell import pipeline, Step
# Define the steps
prepare_environment = Step(
name="prepare_environment",
command="echo 'Preparing environment...'"
)
fetch_data = Step(
name="fetch_data",
command="curl -o data.txt https://example.com/data.txt",
depends_on=[prepare_environment]
)
process_data = Step(
name="process_data",
command="python process_data.py data.txt processed_data.txt",
depends_on=[fetch_data]
)
analyze_data = Step(
name="analyze_data",
command="python analyze_data.py processed_data.txt results.txt",
depends_on=[process_data]
)
cleanup = Step(
name="cleanup",
command="rm data.txt processed_data.txt",
depends_on=[analyze_data],
allow_failure=True # Allow cleanup to fail without affecting pipeline
)
pipeline("example_pipeline", prepare_environment, fetch_data, process_data, analyze_data, cleanup)
Contributing
Contributions are welcome! Please open an issue or submit a pull request on GitHub.
License
This project is licensed under the MIT License.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pipeshell-0.2.0.tar.gz
(5.8 kB
view hashes)
Built Distribution
Close
Hashes for pipeshell-0.2.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7a834e7df6b30bf871cc0f0e57e722789b5e01a8959991566fde279b53c6a3db |
|
MD5 | 94feae73ba5d28c3b787ada63df4cce0 |
|
BLAKE2b-256 | 863b8f62fda9434d44b38103fde5ae9b614a910b724ac15ce1ec61d17520ab14 |