Skip to main content

🗄️ shqaff

PyPI version Python versions Tests License: MIT

A lightweight, PostgreSQL-backed task queue for Python. Uses SQLAlchemy for storage and a finite state machine for reliable task lifecycle management.

Features

  • PostgreSQL-backed persistence with row-level locking (SELECT ... FOR UPDATE SKIP LOCKED)
  • Finite state machine for task transitions: pending -> in_progress -> done | failed
  • Configurable retry logic with max retry limits
  • Simple consumer registration pattern
  • Minimal dependencies: SQLAlchemy, psycopg2, transitions

Installation

pip install shqaff

For development:

pip install shqaff[dev]

Quick Start

1. Start PostgreSQL

Using Docker:

docker run -d --name shqaff-db \
  -e POSTGRES_DB=shqaff \
  -e POSTGRES_USER=postgres \
  -e POSTGRES_PASSWORD=postgres \
  -p 5432:5432 \
  postgres:15

2. Define a Consumer

A consumer is a class that processes tasks. Subclass Consumer and implement the name property and run() method:

from shqaff.consumer import Consumer
from shqaff.registry import register_consumer

class EmailConsumer(Consumer):

    @property
    def name(self) -> str:
        return "send_email"

    def run(self, payload: dict) -> None:
        to = payload["to"]
        subject = payload["subject"]
        body = payload["body"]
        print(f"Sending email to {to}: {subject}")
        # ... actual email sending logic here

register_consumer(EmailConsumer)

3. Create Tasks

from shqaff.db import init_db, SessionLocal
from shqaff.producer import create_task

# Initialize the database tables
init_db()

db = SessionLocal()

# Enqueue a task
create_task(
    db=db,
    task_name="welcome_email",
    consumer="send_email",
    payload={
        "to": "user@example.com",
        "subject": "Welcome!",
        "body": "Thanks for signing up.",
    },
    max_retries=3,
)

4. Process Tasks

from shqaff.event_loop import process_tasks

# Poll the database and process pending tasks
# This runs an infinite loop with a configurable interval
process_tasks(db=db, poll_interval=2, batch_size=10)

Single-batch Processing

For cron-style execution, process one batch and exit:

from shqaff.event_loop import process_once

process_once(db=db, batch_size=10)

Full Working Example

from dataclasses import asdict, dataclass

from shqaff.consumer import Consumer
from shqaff.db import init_db, SessionLocal
from shqaff.event_loop import process_tasks
from shqaff.producer import create_task
from shqaff.registry import register_consumer


@dataclass
class OrderPayload:
    order_id: int
    items: list[str]


class OrderConsumer(Consumer):

    @property
    def name(self) -> str:
        return "process_order"

    def run(self, payload: dict) -> None:
        order = OrderPayload(**payload)
        print(f"Processing order #{order.order_id}: {order.items}")


if __name__ == "__main__":
    init_db()
    register_consumer(OrderConsumer)

    db = SessionLocal()
    create_task(
        db=db,
        task_name="new_order",
        consumer="process_order",
        payload=asdict(OrderPayload(order_id=42, items=["widget", "gadget"])),
    )

    # Process tasks every 2 seconds
    process_tasks(db=db, poll_interval=2)

Repository Layer

TaskRepository is a thin, engine-agnostic CRUD layer over the task model. It holds no connection of its own — it operates on the Session you pass in, so it works identically against PostgreSQL in production and SQLite in tests:

from shqaff import SessionLocal, TaskRepository, TaskStatus

repo = TaskRepository(SessionLocal())

task = repo.create(task_name="welcome_email", consumer="send_email", payload={"to": "a@b.c"})
repo.get(task.id)
repo.list(status=TaskStatus.PENDING, limit=20)
repo.count(status=TaskStatus.PENDING)
repo.set_status(task.id, TaskStatus.DONE)
repo.delete(task.id)

Command-Line Interface

Installing the package exposes a shqaff command with full CRUD over the queue:

shqaff init-db
shqaff create --task-name welcome_email --consumer send_email --payload '{"to": "a@b.c"}'
shqaff list --status pending
shqaff get 1
shqaff update 1 --status done
shqaff delete 1

The database URL is resolved from the --database-url option, the SHQAFF_DATABASE_URL environment variable, or the SHAQAFF_DB_* configuration variables, in that order.

Task Lifecycle

Tasks follow a strict state machine:

pending ──start()──> in_progress ──succeed()──> done
                          │
                          └──fail()──> failed
  • pending: Task is queued, waiting to be picked up
  • in_progress: A consumer is currently executing the task
  • done: Task completed successfully
  • failed: Task exceeded max retries and is permanently failed

When a task raises an exception during processing, it is retried (reset to pending) until max_retries is reached, at which point it transitions to failed.

Configuration

Database connection is configured via environment variables:

Variable Default Description
SHAQAFF_DB_HOST localhost PostgreSQL host
SHAQAFF_DB_PORT 5432 PostgreSQL port
SHAQAFF_DB_NAME shqaff Database name
SHAQAFF_DB_USER postgres Database user
SHAQAFF_DB_PASS (empty) Database password

Docker

Run the full stack with Docker Compose:

docker compose up -d db       # Start PostgreSQL
docker compose up shqaff      # Run the demo app
docker compose run test       # Run the test suite

Testing

The suite spans unit, integration, fuzz (Hypothesis) and end-to-end layers. The repository, CLI and lifecycle tests run against in-memory SQLite and need no PostgreSQL; the legacy DB-backed tests (test_smoke, test_event_loop, test_retry_logic, test_fail_after_max_retries) require a running instance.

pip install -e ".[dev]"

# Runs everywhere, no database required:
pytest tests/test_repository.py tests/test_repository_fuzz.py \
       tests/test_cli.py tests/test_e2e.py tests/test_package.py \
       tests/test_task_fsm.py tests/test_invalid_fsm_transition.py

# Full suite (needs a running PostgreSQL instance):
pytest tests/

License

MIT

Metadata

Release files for shqaff 0.0.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for shqaff 0.0.3
File Size Uploaded Provenance Actions
shqaff-0.0.3.tar.gz 18.9 kB Without provenance Details for shqaff-0.0.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for shqaff 0.0.3
File Interpreter ABI Platform Actions
shqaff-0.0.3-py3-none-any.whl Python 3 none any Details for shqaff-0.0.3-py3-none-any.whl

Total release size: 32.2 kB

Release files / shqaff-0.0.3.tar.gz

Download URL shqaff-0.0.3.tar.gz
Size 18.9 kB
Tags Source
SHA-256 checksum
How to use checksums
0da22a766a1b62dba040b8ee7767b6ad869da3c5f4b323e2787197880d204fcd
BLAKE2b-256 checksum
How to use checksums
5691171d560877d57ba3a65b7abb05fb4f01759cfbe844d99d971f9de7c641ab
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.5

Release files / shqaff-0.0.3-py3-none-any.whl

Download URL shqaff-0.0.3-py3-none-any.whl
Size 13.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
8e37fd9274824a2d25c102dcd421aafb5b496cb718308775c66848e6bdb60b8f
BLAKE2b-256 checksum
How to use checksums
f9ebba251c27313212e7aa7b675fa6c5df23c7c39f7b35997d02b6bf819f7508
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.5

Security for shqaff 0.0.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page