django-vtasks 3.0.0

A very fast valkey/postgres django tasks backend.

django-vtasks

Valkey Tasks. Very Fast Tasks.

From the team at GlitchTip, django-vtasks is a lightweight, async-first task queue for Django 6.0+.

Status: Newly feature complete. Beta quality. Use and report bugs.

Why django-vtasks?

• Async-first - Native asyncio worker for high-performance I/O
• Flexible backends - Start with Postgres, scale to Valkey without rewriting code
• Lightweight - Minimal dependencies, modern codebase
• Embedded mode - Run tasks in your ASGI server or as standalone workers

Features

• Dual backends: Database (Postgres/SQLite/MySQL) and Valkey (Redis-compatible)
• Scheduled tasks with cron syntax
• Delayed tasks (run_after)
• Unique tasks (Mutex and Throttle patterns)
• Per-queue concurrency limits (isolate heavy tasks from cheap ones in one worker)
• Batch processing for high-throughput queues
• Prometheus metrics
• Django admin interface for task management

Requirements

• Python 3.12+
• Django 6.0+
• Valkey 7+ (or Redis 7+) for Valkey backend

Quick Start

pip install django-vtasks

# For Valkey backend support:
pip install "django-vtasks[valkey]"

# settings.py
INSTALLED_APPS = ["django_vtasks", "django_vtasks.db"]

TASKS = {
    "default": {
        "BACKEND": "django_vtasks.backends.db.DatabaseTaskBackend",
    }
}

# myapp/tasks.py
from django_vtasks import task

@task
def send_email(user_id):
    # Your task logic
    pass

# In your views
send_email.enqueue(user_id)
# or async
await send_email.aenqueue(user_id)

# Run the worker
python manage.py runworker

Queue configuration

VTASKS_QUEUES declares the queues a worker consumes. Use a plain list for the simple case, or a dict to give each queue its own options:

# settings.py
# Every concurrency limit is PER WORKER (per process). The fleet-wide cap is
# the limit times your worker count.
VTASKS_CONCURRENCY = 50            # global pool / default per-queue limit, per worker

VTASKS_QUEUES = {
    "default": {},                              # shares the global pool
    "cold_storage": {"worker_concurrency": 3},  # at most 3 at once *per worker*
    "emails": {"batch": {"count": 100, "timeout": 5.0}},
}

# simple form still works:
# VTASKS_QUEUES = ["default", "cold_storage"]

@task(queue_name="cold_storage")
def compact_parquet(org_id):
    ...

Per-queue concurrency. Because vtasks is async-first, a high VTASKS_CONCURRENCY is ideal for cheap I/O-bound tasks — but a few heavy CPU/RAM-bound tasks (analytics, image processing, data exports) at that same concurrency can exhaust memory or a connection pool. A queue with its own worker_concurrency gets a dedicated semaphore; every other queue shares the global VTASKS_CONCURRENCY pool, so a saturated capped queue never blocks the rest. Every limit is per-worker (per-process) — the right scope for bounding per-pod resources like memory or connections — so the fleet-wide ceiling is the limit times your worker count (e.g. worker_concurrency: 3 across 4 workers allows up to 12 concurrent cold_storage tasks cluster-wide). The key name spells this out so it's unambiguous where it's set.

Batching. Declare a batch option on a queue (see emails above) to collect up to count tasks (waiting at most timeout seconds) and hand them to the task as a list.

Delayed tasks

Pass run_after to defer execution until a given time (it's a "not before" guarantee, accurate to about a second — not a precise scheduler):

from datetime import timedelta
from django.utils import timezone

await send_email.using(run_after=timezone.now() + timedelta(minutes=10)).aenqueue(user_id)

Enqueuing from other languages

Valkey is the broker and the integration boundary: any producer that speaks the wire protocol can enqueue tasks a vtasks worker will run — no vtasks dependency required (e.g. a Rust hot path pushing straight to Valkey). The contract — key layout, payload schema, priority, and unique/delayed semantics — is specified in PROTOCOL.md.

Performance

Benchmarks simulate async Django views dispatching tasks — the real ASGI use case.

Scenario (Sleep 10ms)	Enqueue (ops/s)	Process (ops/s)	Peak RSS (MB)	Valkey Conns
VTasks	5,203	3,796	76	3
Celery Threads	2,228	894	123	11
RQ	436	25	170	4

VTasks: 4x faster processing, 2x faster enqueue, 38% less memory vs Celery.

See Benchmarks for full methodology, cloud latency results, and how to reproduce.

Documentation

Full documentation is available at django-vtasks.glitchtip.com

• Getting Started
• Guide - Unique tasks, batching, scheduling, and more
• Configuration - All settings reference
• Deployment - Standalone workers, embedded mode, Kubernetes
• Benchmarks - Performance comparison with Celery

Contributing

We welcome contributions! Please see CONTRIBUTING.md for details.

---

Built by the GlitchTip team.