This release is a pre-release and may not be stable for production use.
TaskForge
Background jobs for Python, stored in your Postgres database. No Redis, no broker.
Status: alpha (0.2.0.dev). The API will change before 1.0. Pre-releases are on PyPI as
taskforge-queue(pip install --pre taskforge-queue); the first stable release is 0.2.0. Seedocs/revamp/for the audit, spec and roadmap.
Some work is too slow to do inside a web request: sending an email, resizing an image, generating a report. TaskForge moves it to a background worker. Jobs are rows in a queue table in your Postgres database, so they survive restarts, and a worker claims each one with SELECT … FOR UPDATE SKIP LOCKED, which means you can run as many workers as you like.
Features
- Postgres-backed queue. Jobs are durable and safe to consume from many workers at once.
@registerdecorator. Any function can be a job.- Retries with exponential backoff, then a dead-letter state for jobs that keep failing, with one-click replay.
- Scheduled jobs. Run no earlier than a given time.
- Tags. Attach key/value metadata to jobs and filter by it.
- Worker registry with heartbeats. See which workers are online, offline or lost.
- Web dashboard (FastAPI + HTMX): overview, job search, job details, replay.
- Prometheus metrics exported by each worker.
- CLI for stats, history, workers, dead-letter and replay.
- Graceful shutdown. On Ctrl+C, SIGTERM or Ctrl+Break, the worker finishes the current job first.
Not implemented yet: reclaiming jobs from crashed workers, concurrency within a worker, priorities, cron schedules, dashboard authentication. See the roadmap.
Requirements
- Python 3.10+
- PostgreSQL 13+. SQLite works for the dashboard and CLI but not for workers, which need
SKIP LOCKED.
Installation
pip install --pre taskforge-queue
Or from source, for development:
git clone https://github.com/B1CA16/taskforge.git
cd taskforge
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install -e ".[dev]"
macOS / Linux
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
Quick start
1. Point TaskForge at a database. Copy .env.example to .env and set the URL, or set it for the current PowerShell session:
$env:TASKFORGE_DATABASE_URL = "postgresql+psycopg://postgres:postgres@127.0.0.1:5435/postgres"
No Postgres yet? See docs/DOCKER_PG_SETUP.md.
2. Create the tables (safe to re-run; it never drops anything):
python -m taskforge.cli.main init-db
3. Define jobs. From demo_client/my_sample_jobs.py:
from taskforge.jobs.registry import register
@register("add_numbers")
def add_numbers(a: int, b: int, logger=None):
if logger:
logger.info(f"Adding {a} and {b}")
return a + b
4. Start a worker, in terminal 1:
python demo_client\run_worker.py
5. Enqueue jobs, in terminal 2:
python demo_client\enqueue_job.py
The worker picks the jobs up immediately. One of the demo jobs fails on purpose, so you'll also see a retry being scheduled and a job moving to the dead-letter queue.
Usage
Defining jobs
- Decorate a function with
@register("job_type"). Each name must be unique; registering a different function under an existing name raisesValueError. - A
dictpayload is passed as keyword arguments, and alistpayload as positional arguments. - If the function has a
loggerparameter, TaskForge injects a logger that addsjob_id,job_typeandworker_idto every record. - The return value is stored as the job's
resultand must be JSON-serializable. - Jobs can run more than once (at-least-once delivery), so make them idempotent.
Enqueuing jobs
from datetime import datetime, timedelta, timezone
from taskforge.task_queue.db import enqueue_single_job
enqueue_single_job(
"add_numbers", # job_type used in @register
{"a": 1, "b": 2}, # dict -> kwargs, list -> args
queue_name="default_queue",
max_attempts=5, # overrides DEFAULT_MAX_ATTEMPTS
scheduled_at=datetime.now(timezone.utc) + timedelta(minutes=5), # naive = UTC
tags={"team": "billing"},
)
Running workers
from taskforge.config.logging import setup_logging
from taskforge.worker.worker import Worker
import my_jobs # importing the module registers the jobs
setup_logging() # TaskForge never configures logging on its own
Worker(queues=["default_queue"]).run()
Worker options: queues, poll_interval (seconds to wait when idle, default 1), enable_metrics, metrics_port (default TASKFORGE_METRICS_PORT or 9464), handle_signals.
A failed job is retried after 10 × 2^attempt seconds (20 s, 40 s, …) and moves to dead after max_attempts.
Job states
| Status | Meaning |
|---|---|
pending |
Waiting for its first run |
running |
Claimed by a worker |
failed |
Last attempt failed; retry scheduled at scheduled_at |
done |
Finished successfully |
dead |
Out of attempts; inspect it, then replay it |
Configuration
| Variable | Default | Purpose |
|---|---|---|
TASKFORGE_DATABASE_URL (or DATABASE_URL) |
none (required) | SQLAlchemy URL, e.g. postgresql+psycopg://user:pw@host:5432/db. A .env file in the working directory is read if neither is set. |
DEFAULT_MAX_ATTEMPTS |
3 |
Attempts per job (including the first run) |
TASKFORGE_METRICS_PORT |
9464 |
Port for each worker's Prometheus exporter |
CLI
python -m taskforge.cli.main init-db # create missing tables
python -m taskforge.cli.main stats # job counts per queue and status
python -m taskforge.cli.main history --status dead --tag team=billing --limit 50
python -m taskforge.cli.main workers # registered workers and heartbeat status
python -m taskforge.cli.main dead-letter # dead jobs with their last error
python -m taskforge.cli.main replay <job_id> # re-enqueue a dead job (once; --force to repeat)
Dashboard
python demo_client\run_dashboard.py
Open http://127.0.0.1:8000. The JSON API is at /api/* and the interactive docs at /docs.
⚠️ The dashboard has no authentication yet. Keep it on
127.0.0.1or behind a proxy that handles auth.
Metrics
Each worker serves Prometheus metrics at http://<host>:9464/metrics:
taskforge_jobs_processed_total{queue,job_type,status}taskforge_jobs_enqueued_total{queue,job_type}(counted in the process that enqueues)taskforge_job_execution_duration_secondstaskforge_queue_depth{queue,status}taskforge_active_workers{queue}
If the port is already taken (for example, by a second worker on the same machine), the worker logs a warning and keeps running without an exporter.
Development
The test suite runs against a disposable Postgres in Docker (port 5436, data in memory):
docker compose up -d test-db
pytest
The suite drops all tables, so it refuses to run unless the database name contains test. Point it elsewhere with TASKFORGE_TEST_DATABASE_URL. See TESTING_STRATEGY.md.
License
MIT. See LICENSE.
Metadata
Release files for taskforge-queue 0.2.0.dev0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| taskforge_queue-0.2.0.dev0.tar.gz | 74.8 kB | Without provenance | Details for taskforge_queue-0.2.0.dev0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| taskforge_queue-0.2.0.dev0-py3-none-any.whl | Python 3 | none | any | Details for taskforge_queue-0.2.0.dev0-py3-none-any.whl |
Total release size: 114.7 kB
Release files / taskforge_queue-0.2.0.dev0.tar.gz
| Download URL | taskforge_queue-0.2.0.dev0.tar.gz |
|---|---|
| Size | 74.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4a6890dc21e5f0592967252d5544dbe66697d892ac7ec65b2a5399a19e65f45d
|
|
BLAKE2b-256 checksum How to use checksums |
70a2fdcc43ba03512c50e6889c008264242e30dd21d5c91bde437b81bd5ad953
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.5
|
Release files / taskforge_queue-0.2.0.dev0-py3-none-any.whl
| Download URL | taskforge_queue-0.2.0.dev0-py3-none-any.whl |
|---|---|
| Size | 39.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
bd73065cd86805cb61d6c9610ebc1b900078c56d59990244fbef07f39e016b48
|
|
BLAKE2b-256 checksum How to use checksums |
3fc77e2738935e4a4d3d8430babe0cd2cbf9a05e6643ec43481e03e1b4fbca70
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.5
|
Security for taskforge-queue 0.2.0.dev0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.