pipeweave 0.1.0

No project description provided

Pipeweave

A flexible Python data pipeline library that makes it easy to construct and run custom data pipelines using a finite state machine approach.

Project Goal

I have tried some popular Python data pipeline libraries, and have found them all to be a little hard to use for custom use cases. The goal of this project is to create a pipeline library that avoids some of the common pitfalls and allows users to easily construct pipelines using custom functions and run them using a finite state machine.

Features

• 🚀 Simple, intuitive API for creating data pipelines
• 🔄 Built-in state management using finite state machines
• 📦 Easy integration of custom functions
• 💾 Multiple storage backends (SQLite included)
• 🔍 Pipeline status tracking and monitoring
• ⚡ Efficient execution with dependency management

Installation

pip install pipeweave

Quick Start

Here's a simple example that demonstrates how to create and run a pipeline:

pip install pipeweave

Quick Start

Here's a simple example that demonstrates how to create and run a pipeline:

from pipeweave.core import Pipeline

# Create a pipeline
pipeline = Pipeline(name="data_transformer")

# Define some processing functions
def clean_data(data):
    return [x.strip().lower() for x in data]

def filter_empty(data):
    return [x for x in data if x]

# Add steps to the pipeline
pipeline.add_step(
    name="clean_data",
    description="Clean the data",
    function=clean_data,
    inputs=["raw_data"],
    outputs=["cleaned_data"],
)

pipeline.add_step(
    name="filter_empty",
    description="Filter out empty strings",
    function=filter_empty,
    inputs=["cleaned_data"],
    outputs=["filtered_data"],
    dependencies=["clean_data"],
)

# Run the pipeline
data = [" Hello ", "World ", "", " Python "]

results = pipeline.run(data)

print(results)

Core Concepts

Steps

A Step is the basic building block of a pipeline. Each step:

• Has a unique name
• Contains a processing function
• Defines its inputs and outputs
• Can specify dependencies on other steps
• Maintains its own state (IDLE, RUNNING, COMPLETED, ERROR)

Pipeline

A Pipeline is a collection of steps that:

• Manages the execution order based on dependencies
• Handles data flow between steps
• Tracks overall execution state
• Can be saved and loaded using storage backends

Storage Backends

Pipeweave supports different storage backends for persisting pipelines:

• SQLite (included)
• Custom backends can be implemented using the StorageBackend base class

Advanced Usage

Using Storage Backends

# Initialize Storage
storage = SQLiteStorage("pipelines.db")

# Save Pipeline
storage.save_pipeline(pipeline)

# Load Pipeline
pipeline = storage.load_pipeline("data_transformer")

Error Handling

from pipeweave.steps import State

try:
    results = pipeline.run(data)
except Exception as e:
    # Check state of steps
    for step in pipeline.steps.values():
        if step.state == State.ERROR:
            print(f"Step {step.name} failed: {step.error}")

Contributing

Contributions are welcome! This is a new project, so please feel free to open issues and suggest improvements.

For major changes, please open an issue first to discuss what you would like to change.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Project Status

Pipeweave is currently in alpha. While it's functional and tested, the API may change as we gather user feedback and add new features.

Roadmap

• Add a more robust state machine implementation
• Add more storage backends
• Add more detailed monitoring and logging
• Add more testing and CI/CD pipeline