pgskewer
A minimalistic pipeline functionality built on top of pgqueuer, providing enhanced task orchestration capabilities for PostgreSQL-based job queues.
The name "pgskewer" is chosen due to its metaphorical representation of how it "skewers" tasks together in a pipeline; rhymes with "pgqueuer" and contains "EW" (Education Warehouse) in its name.
Features
- Sequential and Parallel Task Pipelines: Define complex workflows with a mix of sequential and parallel task execution
- Result Storage: Automatically store and pass around task results in sequence
- Cancelable Tasks: Gracefully cancel tasks when needed
- Improved Error Handling: Better error propagation and handling in pipelines
- Real-time Log Streaming: Stream logs from blocking functions in real-time
- Type Annotations: Comprehensive typing support for better IDE integration
Installation
uv pip install pgskewer
Requirements
- Python ≥ 3.13
- PostgreSQL database
- Dependencies:
- pgqueuer
- asyncpg
- anyio
- uvloop
Quick Start
import asyncio
import asyncpg
from pgqueuer import Job
from pgqueuer.db import AsyncpgDriver
from pgqueuer.queries import Queries
from pgskewer import ImprovedQueuer, parse_payload, TaskResult
async def main():
# Initialize the queuer with your database connection
connection = await asyncpg.connect("postgresql://user:password@localhost/dbname")
driver = AsyncpgDriver(connection)
pgq = ImprovedQueuer(driver)
# Define some tasks as entrypoints
@pgq.entrypoint("fetch_data")
async def fetch_data(job):
# Fetch some data
return {"data": "example data"}
@pgq.entrypoint("process_data")
async def process_data(job: Job):
# Process the data from the previous step
payload = parse_payload(job.payload)
data = payload["tasks"]["fetch_data"]["result"]["data"]
return {"processed": data.upper()}
@pgq.entrypoint("store_results")
async def store_results(job: Job):
# Store the processed data
payload = parse_payload(job.payload)
processed = payload["tasks"]["process_data"]["result"]["processed"]
# Store the processed data somewhere
return {"status": "completed", "stored": processed}
# Create a pipeline that runs these tasks in sequence
pgq.entrypoint_pipeline(
"my_pipeline",
# start steps as a mix of entrypoint names and function references:
fetch_data,
"process_data",
store_results
)
# Execute the pipeline (empty initial data)
job_id = await pgq.qm.queries.enqueue("my_pipeline", b'')
# when the pipeline completes, pgqueuer_result should have an entry for this job_id:
result: TaskResult = await pgq.result(job_id, timeout=None)
if __name__ == "__main__":
asyncio.run(main())
Advanced Usage
Creating Pipelines with Parallel Tasks
You can define pipelines with a mix of sequential and parallel tasks:
# Define a pipeline with parallel tasks
pipeline = pgq.pipeline([
"task_1", # Run task_1 first
["task_2a", "task_2b"], # Then run task_2a and task_2b in parallel
"task_3" # Finally run task_3 after both task_2a and task_2b complete
])
Register a Pipeline as an Entrypoint
You can register a pipeline as an entrypoint for reuse:
# Register the pipeline as an entrypoint
pgq.entrypoint_pipeline(
"data_processing_pipeline",
"fetch_data",
["validate_data", "normalize_data"],
"store_data"
)
# Now you can enqueue this pipeline like any other task
job_id = await pgq.enqueue("data_processing_pipeline", {"source": "api"})
Running Blocking Functions Asynchronously
pgskewer provides utilities to run blocking functions asynchronously with real-time log streaming:
from pgskewer import unblock
def cpu_intensive_task(data):
# This is a blocking function
print("Processing data...")
result = process_data(data)
print("Processing complete!")
return result
# Run the blocking function asynchronously with log streaming
result = await unblock(cpu_intensive_task, data)
Pipeline Result Structure
The pipeline returns a structured result with information about each task:
{
"initial": {
# The initial payload provided to the pipeline
},
"tasks": {
"task_1": {
"status": "successful",
"ok": true,
"result": {
# Task 1's result data
}
},
"task_2a": {
"status": "successful",
"ok": true,
"result": {
# Task 2a's result data
}
},
# ... other tasks
}
}
Error Handling
If any task in a pipeline fails:
- In sequential execution, the pipeline stops and no further tasks are executed
- In parallel execution, sibling tasks are terminated
License
pgskewer is distributed under the terms of the MIT license.
Credits
Developed by Education Warehouse.
Metadata
Release files for pgskewer 0.4.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| pgskewer-0.4.2.tar.gz | 18.5 kB | Without provenance | Details for pgskewer-0.4.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| pgskewer-0.4.2-py3-none-any.whl | Python 3 | none | any | Details for pgskewer-0.4.2-py3-none-any.whl |
Total release size: 39.0 kB
Release files / pgskewer-0.4.2.tar.gz
| Download URL | pgskewer-0.4.2.tar.gz |
|---|---|
| Size | 18.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
60f10be19173cf77eb1a0531ec9f2878ab0a1c6a7b60d153621cbd8761ad4571
|
|
BLAKE2b-256 checksum How to use checksums |
0812067948761993ebf229ef59f08d79d073400ea8d7e63e5a7326aebc89f07c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.10.4 {"installer":{"name":"uv","version":"0.10.4","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Linux Mint","version":"22.3","id":"zena","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / pgskewer-0.4.2-py3-none-any.whl
| Download URL | pgskewer-0.4.2-py3-none-any.whl |
|---|---|
| Size | 20.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
bbb7b0722b38c89a3136a69ced63f3631561d4b8362b4273748f4819b6091043
|
|
BLAKE2b-256 checksum How to use checksums |
a69c36a109d31e72aa5fcfd2967c2067a301b6ae1773b357769c636e4decb813
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.10.4 {"installer":{"name":"uv","version":"0.10.4","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Linux Mint","version":"22.3","id":"zena","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for pgskewer 0.4.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.