Skip to main content

Pipeline Connector

A powerful Python package for connecting and orchestrating multiple data pipelines with ease.

Features

  • Multi-Pipeline Connection: Connect 2 or more pipelines seamlessly
  • Data Flow Management: Intelligent data routing between connected pipelines
  • Asynchronous Support: Built-in async support for high-performance processing
  • Type Safety: Full type hints and Pydantic validation
  • Extensible Architecture: Easy to extend with custom pipeline implementations
  • Error Handling: Robust error handling and recovery mechanisms
  • Logging: Comprehensive logging for debugging and monitoring

Installation

pip install pipeline-connector

Quick Start

from pipeline_connector import PipelineManager, DataPipeline

# Create pipeline instances
pipeline1 = DataPipeline("data_source")
pipeline2 = DataPipeline("data_processor") 
pipeline3 = DataPipeline("data_sink")

# Create a pipeline manager
manager = PipelineManager()

# Connect pipelines
manager.connect(pipeline1, pipeline2)
manager.connect(pipeline2, pipeline3)

# Execute the connected pipelines
result = await manager.execute(input_data)

Advanced Usage

Custom Pipeline Implementation

from pipeline_connector import BasePipeline
from typing import Any, Dict

class CustomDataProcessor(BasePipeline):
    async def process(self, data: Any) -> Any:
        # Your custom processing logic
        processed_data = self.transform(data)
        return processed_data
    
    def transform(self, data: Any) -> Any:
        # Implement your transformation logic
        return data

Pipeline with Configuration

from pipeline_connector import PipelineManager, PipelineConfig

config = PipelineConfig(
    name="my_pipeline",
    batch_size=100,
    timeout=30,
    retry_attempts=3
)

manager = PipelineManager(config=config)

Error Handling

from pipeline_connector import PipelineManager, PipelineError

try:
    result = await manager.execute(data)
except PipelineError as e:
    print(f"Pipeline error: {e}")
    # Handle the error appropriately

API Reference

PipelineManager

The main class for managing and connecting multiple pipelines.

Methods

  • connect(source_pipeline, target_pipeline): Connect two pipelines
  • disconnect(source_pipeline, target_pipeline): Disconnect pipelines
  • execute(data, **kwargs): Execute the connected pipeline chain
  • get_connections(): Get all current pipeline connections

BasePipeline

Abstract base class for creating custom pipelines.

Methods

  • process(data): Abstract method to implement data processing logic
  • validate_input(data): Validate input data
  • validate_output(data): Validate output data

DataPipeline

Ready-to-use pipeline implementation for common data processing tasks.

Examples

Check out the examples/ directory for more detailed usage examples:

  • basic_usage.py: Simple pipeline connection example
  • async_processing.py: Asynchronous pipeline processing
  • custom_pipeline.py: Creating custom pipeline implementations
  • error_handling.py: Error handling and recovery

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Changelog

Version 0.1.0

  • Initial release
  • Basic pipeline connection functionality
  • Asynchronous support
  • Type safety with Pydantic
  • Comprehensive error handling

Metadata

Release files for pipeline-connector 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pipeline-connector 0.1.0
File Size Uploaded Provenance Actions
pipeline_connector-0.1.0.tar.gz 15.4 kB Without provenance Details for pipeline_connector-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pipeline-connector 0.1.0
File Interpreter ABI Platform Actions
pipeline_connector-0.1.0-py3-none-any.whl Python 3 none any Details for pipeline_connector-0.1.0-py3-none-any.whl

Total release size: 30.3 kB

Release files / pipeline_connector-0.1.0.tar.gz

Download URL pipeline_connector-0.1.0.tar.gz
Size 15.4 kB
Tags Source
SHA-256 checksum
How to use checksums
9bbb0b7e81116cd2a3bc6e00694a878b2f4ca701da9a90007ae871905d20e1b1
BLAKE2b-256 checksum
How to use checksums
0478bf30f06847b6bdde9a18a1a2722328d648d8037c622ee2877e23437c8d55
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/5.0.0 CPython/3.12.3

Release files / pipeline_connector-0.1.0-py3-none-any.whl

Download URL pipeline_connector-0.1.0-py3-none-any.whl
Size 14.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
36558fbb6f58575f97f372f963620bc4520ea583e41ba237a1afea631ae8effc
BLAKE2b-256 checksum
How to use checksums
12215f6f37e870a5537ef60d76190fad7c3ea8e7dd928187cbc8bae1fa6d3185
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/5.0.0 CPython/3.12.3

Security for pipeline-connector 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page