Pipeline Connector
A powerful Python package for connecting and orchestrating multiple data pipelines with ease.
Features
- Multi-Pipeline Connection: Connect 2 or more pipelines seamlessly
- Data Flow Management: Intelligent data routing between connected pipelines
- Asynchronous Support: Built-in async support for high-performance processing
- Type Safety: Full type hints and Pydantic validation
- Extensible Architecture: Easy to extend with custom pipeline implementations
- Error Handling: Robust error handling and recovery mechanisms
- Logging: Comprehensive logging for debugging and monitoring
Installation
pip install pipeline-connector
Quick Start
from pipeline_connector import PipelineManager, DataPipeline
# Create pipeline instances
pipeline1 = DataPipeline("data_source")
pipeline2 = DataPipeline("data_processor")
pipeline3 = DataPipeline("data_sink")
# Create a pipeline manager
manager = PipelineManager()
# Connect pipelines
manager.connect(pipeline1, pipeline2)
manager.connect(pipeline2, pipeline3)
# Execute the connected pipelines
result = await manager.execute(input_data)
Advanced Usage
Custom Pipeline Implementation
from pipeline_connector import BasePipeline
from typing import Any, Dict
class CustomDataProcessor(BasePipeline):
async def process(self, data: Any) -> Any:
# Your custom processing logic
processed_data = self.transform(data)
return processed_data
def transform(self, data: Any) -> Any:
# Implement your transformation logic
return data
Pipeline with Configuration
from pipeline_connector import PipelineManager, PipelineConfig
config = PipelineConfig(
name="my_pipeline",
batch_size=100,
timeout=30,
retry_attempts=3
)
manager = PipelineManager(config=config)
Error Handling
from pipeline_connector import PipelineManager, PipelineError
try:
result = await manager.execute(data)
except PipelineError as e:
print(f"Pipeline error: {e}")
# Handle the error appropriately
API Reference
PipelineManager
The main class for managing and connecting multiple pipelines.
Methods
connect(source_pipeline, target_pipeline): Connect two pipelinesdisconnect(source_pipeline, target_pipeline): Disconnect pipelinesexecute(data, **kwargs): Execute the connected pipeline chainget_connections(): Get all current pipeline connections
BasePipeline
Abstract base class for creating custom pipelines.
Methods
process(data): Abstract method to implement data processing logicvalidate_input(data): Validate input datavalidate_output(data): Validate output data
DataPipeline
Ready-to-use pipeline implementation for common data processing tasks.
Examples
Check out the examples/ directory for more detailed usage examples:
basic_usage.py: Simple pipeline connection exampleasync_processing.py: Asynchronous pipeline processingcustom_pipeline.py: Creating custom pipeline implementationserror_handling.py: Error handling and recovery
Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
License
This project is licensed under the MIT License - see the LICENSE file for details.
Changelog
Version 0.1.0
- Initial release
- Basic pipeline connection functionality
- Asynchronous support
- Type safety with Pydantic
- Comprehensive error handling
Metadata
Release files for pipeline-connector 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| pipeline_connector-0.1.0.tar.gz | 15.4 kB | Without provenance | Details for pipeline_connector-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| pipeline_connector-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for pipeline_connector-0.1.0-py3-none-any.whl |
Total release size: 30.3 kB
Release files / pipeline_connector-0.1.0.tar.gz
| Download URL | pipeline_connector-0.1.0.tar.gz |
|---|---|
| Size | 15.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
9bbb0b7e81116cd2a3bc6e00694a878b2f4ca701da9a90007ae871905d20e1b1
|
|
BLAKE2b-256 checksum How to use checksums |
0478bf30f06847b6bdde9a18a1a2722328d648d8037c622ee2877e23437c8d55
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.12.3
|
Release files / pipeline_connector-0.1.0-py3-none-any.whl
| Download URL | pipeline_connector-0.1.0-py3-none-any.whl |
|---|---|
| Size | 14.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
36558fbb6f58575f97f372f963620bc4520ea583e41ba237a1afea631ae8effc
|
|
BLAKE2b-256 checksum How to use checksums |
12215f6f37e870a5537ef60d76190fad7c3ea8e7dd928187cbc8bae1fa6d3185
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.12.3
|
Security for pipeline-connector 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.