The Pipeline Design Pattern realization in python.
Project description
The Pipeline Design Pattern
tpdp it is the Pipeline Design Pattern realization in Python.
Quickstart
Install with pip
pip install tpdp
and creating your first pipeline
from tpdp import Pipeline, State, Step
class SimpleState(State):
pipeline_name: str
step_1_count: int = 0
step_2_count: int = 0
init_state = SimpleState(pipeline_name="simple_pipeline")
pipeline = Pipeline(name="simple_pipeline")
class Step1(Step):
def run(self, state: SimpleState, **kwargs) -> SimpleState:
state.step_1_count += 1
self.logger.info("step_1_count increased by 1")
return state
class Step2(Step):
def run(self, state: SimpleState, **kwargs) -> SimpleState:
state.step_2_count += 1
self.logger.info("step_2_count increased by 1")
return state
step_1 = Step1(step_name="step_1")
step_2 = Step2(step_name="step_2")
pipeline.registry_step(step_1)
pipeline.registry_step(step_2)
pipeline.registry_step(step_1)
pipeline.registry_step(step_2)
pipeline.registry_step(step_1)
pipeline.registry_step(step_2)
pipeline.registry_step(step_1)
result = pipeline.run(init_state=init_state)
final_state = pipeline.get_state()
print(f"final_state.step_1_count = {final_state.step_1_count}")
print(f"final_state.step_2_count = {final_state.step_2_count}")
You will see the following logs
[...: INFO] Pipeline: Step registered: step_name="step_1"
[...: INFO] Pipeline: Step registered: step_name="step_2"
[...: INFO] Pipeline: Step registered: step_name="step_1"
[...: INFO] Pipeline: Step registered: step_name="step_2"
[...: INFO] Pipeline: Step registered: step_name="step_1"
[...: INFO] Pipeline: Step registered: step_name="step_2"
[...: INFO] Pipeline: Step registered: step_name="step_1"
[...: INFO] Pipeline: Pipeline start: pipeline_name="simple_pipeline", start_at="..."
[...: INFO] Pipeline: Step start: step_name="step_1"
[...: INFO] Step1: step_1_count increased by 1
[...: INFO] Pipeline: Step finish: step_name="step_1", step_duration="0.000024"
[...: INFO] Pipeline: Step start: step_name="step_2"
[...: INFO] Step2: step_2_count increased by 1
[...: INFO] Pipeline: Step finish: step_name="step_2", step_duration="0.000020"
[...: INFO] Pipeline: Step start: step_name="step_1"
[...: INFO] Step1: step_1_count increased by 1
[...: INFO] Pipeline: Step finish: step_name="step_1", step_duration="0.000016"
[...: INFO] Pipeline: Step start: step_name="step_2"
[...: INFO] Step2: step_2_count increased by 1
[...: INFO] Pipeline: Step finish: step_name="step_2", step_duration="0.000016"
[...: INFO] Pipeline: Step start: step_name="step_1"
[...: INFO] Step1: step_1_count increased by 1
[...: INFO] Pipeline: Step finish: step_name="step_1", step_duration="0.000016"
[...: INFO] Pipeline: Step start: step_name="step_2"
[...: INFO] Step2: step_2_count increased by 1
[...: INFO] Pipeline: Step finish: step_name="step_2", step_duration="0.000016"
[...: INFO] Pipeline: Step start: step_name="step_1"
[...: INFO] Step1: step_1_count increased by 1
[...: INFO] Pipeline: Step finish: step_name="step_1", step_duration="0.000023"
[...: INFO] Pipeline: Pipeline finish: pipeline_name="simple_pipeline", finish_at="...", duration="0.0005033016204833984"
final_state.step_1_count = 4
final_state.step_2_count = 3
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
tpdp_pydantic_v2-1.0.0.tar.gz
(15.0 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file tpdp_pydantic_v2-1.0.0.tar.gz.
File metadata
- Download URL: tpdp_pydantic_v2-1.0.0.tar.gz
- Upload date:
- Size: 15.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.13.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
9f28915822b51b0eb4f8d9cffb6ef9e7d959938cbd6a9b5c0033bb964048ebe0
|
|
| MD5 |
2ccb60729d4d2bcfb40bd47e5755e635
|
|
| BLAKE2b-256 |
61c51df9146be957f080d18a3384bc51f175fe4ba77386dd0a4e3e29f3ef839d
|
File details
Details for the file tpdp_pydantic_v2-1.0.0-py3-none-any.whl.
File metadata
- Download URL: tpdp_pydantic_v2-1.0.0-py3-none-any.whl
- Upload date:
- Size: 5.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.13.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
6a009264776a4a60ac4fbb78256895fafd37e4e99537a7df386d6720b991a9cc
|
|
| MD5 |
b07df183b9bcb0e9709ac71a82f1cf07
|
|
| BLAKE2b-256 |
58af20fdda72c27cfea38e95b91c708197b4a6f637517a8d17b94ef57a348a68
|
