Polidoro Pipeline

A powerful Python library for parallel data processing through a sequence of steps.

Python Versions

🚀 Overview

Polidoro Pipeline is a Python library that simplifies parallel data processing through a sequence of steps. It automatically handles parallelization using Python's ThreadPoolExecutor, making it easy to process both single values and lists of values efficiently.

✨ Features

• 🔄 Process data through a sequence of steps
• ⚡ Automatic parallelization of processing
• 🧩 Simple and intuitive API
• 🔌 Easy to integrate with existing code
• 📦 Handles both single values and lists of values

📋 Installation

pip install polidoro_pipeline

🔧 Usage

Basic Example

from ppipeline import Pipeline

# Define processing steps
def add_1(x):
    return x + 1

def multiply_by_2(x):
    return x * 2

# Create a pipeline with steps
pipeline = Pipeline([add_1, multiply_by_2])

# Process a single value
result = list(pipeline.run(1))  # [4]

# Process multiple values in parallel
results = list(pipeline.run([1, 2, 3]))  # [4, 6, 8]

Adding Steps Incrementally

pipeline = Pipeline()
pipeline.add_step(add_1)
pipeline.add_step(multiply_by_2)
result = list(pipeline.run(2))  # [6]

Controlling Thread Count

# Limit the number of worker threads
pipeline = Pipeline([add_1, multiply_by_2], thread_count=4)
results = list(pipeline.run([1, 2, 3, 4, 5]))

🧠 How It Works

1. The Pipeline class takes a list of callable functions as steps
2. When you call run() with input data, each item is processed through all steps in sequence
3. If the input is a list, items are processed in parallel using ThreadPoolExecutor
4. Each step can return a single value or a list of values
5. If a step returns multiple values (as a list), each value is processed independently in subsequent steps

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add some amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.