django-vpg 0.1.0

Rust-powered PostgreSQL driver and Django database backend.

django-vpg

Status: Experimental. Freshly extracted from glitchtip-rust, where the driver runs GlitchTip in production — but this standalone package is new, not yet on PyPI, and its API and packaging may change without notice. Evaluate freely; don't bet production on it yet.

A Rust-powered PostgreSQL driver and Django database backend.

django-vpg replaces the connection/cursor IO layer of Django's stock PostgreSQL backend with a Rust core (tokio-postgres + deadpool, via PyO3): swap one ENGINE line, keep your schema, migrations, and ORM code unchanged.

Part of the django-v* family — django-vcache, django-vtasks, django-vpg — small, fast, Rust-core Django infrastructure maintained by the GlitchTip team. django-vcache and django-vtasks are battle-tested in production at glitchtip.com; django-vpg is the newest member. (The v originally stood for Valkey; with three packages it now just marks the family.)

Why

• CPU and RSS. Protocol parsing, row decoding, and type marshalling run in Rust; only final Python values are materialized — intermediate protocol buffers and COPY encoding never traverse the Python heap. In the GlitchTip team's A/B benchmarks against psycopg (production-shaped ingest workloads, replayed locally), the driver holds resident memory flat under heavy-tailed payloads where libpq's buffers ratchet (2.5 vs 21 MB growth per 10k events), and wins per-op CPU on jsonb reads, pipelined batch inserts, and COPY event batches.
• fetch_models fusion. Row → model-instance materialization can run as a single driver call instead of a per-row Python loop.
• COPY at native speed. COPY FROM STDIN with batch text-format encoding in Rust (encode_copy_chunk).
• psycopg-shaped errors. Full PEP 249 exception hierarchy with sqlstate and Diagnostics, so existing error-handling code ports cleanly.
• Server-side cancellation. Cancelled queries fire a PostgreSQL CancelRequest on a fresh connection and the pooled connection returns clean — no poisoned pool state, no abandoned server work.
• Real pooling. deadpool checkout/return with warmup, wait timeouts, and jittered max-lifetime recycling; prepared-statement cache with psycopg3-style admission (prepare_threshold / prepared_max); pgbouncer mode (OPTIONS={"pgbouncer": True}, which disables prepared statements).
• libpq-parity TLS. sslmode (disable/allow/prefer/require/ verify-ca/verify-full), sslrootcert, client certificates.

Install

pip install django-vpg          # sync ORM + async driver API
pip install django-vpg[async]   # + async ORM via django-async-backend (Django 6+)

Note: psycopg (the library) must be installed — it is a declared dependency. Django's stock postgresql backend, which django-vpg subclasses for schema/introspection/operations, imports it at load time. The IO path never uses it.

Quickstart

DATABASES = {
    "default": {
        "ENGINE": "django_vpg.backend",
        "NAME": "mydb",
        "USER": "postgres",
        "PASSWORD": "...",
        "HOST": "db.example.com",
        "PORT": "5432",
        # Optional driver OPTIONS (libpq-parity naming; psycopg3-compatible
        # pool dict):
        # "OPTIONS": {
        #     "pool": {"max_size": 20, "min_size": 2},   # default max_size: 10
        #     "sslmode": "verify-full",
        #     "sslrootcert": "/path/ca.pem",
        # },
    }
}

Supported OPTIONS keys: pool (psycopg3-compatible dict — max_size, min_size, timeout, max_lifetime, max_idle, name — or False for a dedicated single-slot connection per Django connection, e.g. under test runners), pool_size (shorthand for pool={"max_size": N}; default 10), pgbouncer, isolation_level, application_name, connect_timeout, keepalives, keepalives_idle, prepare_threshold, prepared_max, sslmode, sslrootcert, sslcert, sslkey, and server_settings. Unknown keys warn once and are dropped.

The async story (three layers)

1. Sync ORM — stock Django, zero extra deps. Works everywhere Django works.
2. Async driver API — real asyncio, works anywhere. The raw driver (django_vpg._driver.RustPgDriver) exposes await-able query, execute, transaction, and COPY operations with no dependency on any async framework beyond asyncio itself; driver awaitables interoperate with gather, wait_for, shield, and cancellation. (django_vpg.dbapi itself is the sync DB-API 2.0 shim the Django engine builds on.)
3. Async ORM — via django-async-backend. Installing the django-vpg[async] extra activates AsyncDatabaseWrapper, which plugs into the async_connections framework. Without it, the backend cleanly degrades to sync-only (no ImportError).

Unifying Django's connections and async_connections (transaction management and isolation across both worlds) is active upstream work in django-async-backend that this package tracks closely.

The tokio runtime

The driver runs on a fork-safe shared tokio runtime owned by the package (1 worker thread by default; VPG_TOKIO_WORKER_THREADS overrides). Prefork servers are safe: the runtime is rebuilt per child on first use.

Rust embedders get the same hook django-vcache offers: link vpg-pyo3 as an rlib, re-export its pyclasses into your own pymodule, and install a process-wide runtime with vpg_core::set_runtime_provider(my_get_runtime) so one tokio pool serves every driver in the process. This mirrors how glitchtip-rust embeds its driver today (rlib link + re-exported pyclasses + one shared runtime); gt_rust will switch to linking vpg-pyo3 and installing its runtime via this hook.

Status

Extracted from glitchtip-rust (v0.6.1), where this driver is the only database engine in GlitchTip's backend (as of July 2026). Pre-1.0 while the standalone packaging settles; the driver core predates this package and carries its full test suite (including an in-process PG wire-server harness) with it.

Known naming wart: the vendored tokio-postgres buffer-cap patch reads the GT_PG_BUF_CAP_BYTES env var (512 KiB default retained-buffer cap); it will be renamed with a compatibility window before 1.0.

Development

./scripts/check.sh        # fmt + clippy -D warnings + cargo test + ruff
                          # (includes tests/wire.rs — an in-process scripted
                          #  PG wire server; no live postgres needed)

# Python suite (needs maturin + a live postgres for the marked tests):
python -m venv .venv && source .venv/bin/activate
pip install maturin pytest pytest-asyncio django psycopg django-async-backend
maturin develop --release
VPG_TEST_DSN=postgres://postgres@localhost:5432/postgres python -m pytest tests/

The workspace is two crates: crates/vpg-core (pure Rust, no pyo3, cargo test-able) and crates/vpg-pyo3 (thin bindings, cdylib + rlib). vendor/tokio-postgres carries a minimal retained-buffer-capacity patch — see vendor/tokio-postgres/GT_PATCH.md for rationale and the re-audit procedure on version bumps.

License

MIT