django-vpg

A Rust-powered PostgreSQL driver and Django database backend.

• A single connection pool used for both sync and async (asyncio) usage
• Very slightly better cpu/memory effeciency, especially with large INSERTs, when compared to psycopg
• Callable from Python or Rust with single connection pool
• Drop in compatibility "ENGINE": "django_vpg.backend"
• Supports django-async-backend for real asyncio usage

Build with tokio-postgres, deadpool, PyO3.

Status: Beta quality. Mostly feature complete. Report bugs. Use in production at your own risk.

Documentation: https://django-vpg.glitchtip.com

Part of the django-v* family — django-vcache, django-vtasks, django-vpg — small, fast, Rust-core Django infrastructure maintained by the GlitchTip team.

Why

• CPU and RSS. Protocol parsing, row decoding, and type marshalling run in Rust; only final Python values are materialized — intermediate protocol buffers and COPY encoding never traverse the Python heap. In the GlitchTip team's A/B benchmarks against psycopg (production-shaped ingest workloads, replayed locally), the driver holds resident memory flat under heavy-tailed payloads where libpq's buffers ratchet (2.5 vs 21 MB growth per 10k events), and wins per-op CPU on jsonb reads, pipelined batch inserts, and COPY event batches.
• fetch_models fusion. Row → model-instance materialization can run as a single driver call instead of a per-row Python loop.
• COPY at native speed. COPY FROM STDIN with batch text-format encoding in Rust (encode_copy_chunk).
• psycopg-shaped errors. Full PEP 249 exception hierarchy with sqlstate and Diagnostics, so existing error-handling code ports cleanly.
• Server-side cancellation. Cancelled queries fire a PostgreSQL CancelRequest on a fresh connection and the pooled connection returns clean — no poisoned pool state, no abandoned server work.
• Real pooling. deadpool checkout/return with warmup, wait timeouts, and jittered max-lifetime recycling; prepared-statement cache with psycopg3-style admission (prepare_threshold / prepared_max), off by default like Django's postgresql backend — opt in with OPTIONS={"prepare_threshold": 5}, and leave it off behind a transaction-mode PgBouncer without max_prepared_statements.
• libpq-parity TLS. sslmode (disable/allow/prefer/require/ verify-ca/verify-full), sslrootcert, client certificates.

Install

pip install django-vpg            # sync ORM + async driver API
pip install "django-vpg[async]"   # + async ORM via django-async-backend

Requires Django >= 6.1 and Python >= 3.12.

Wheels cover Linux (manylinux + musllinux, x86_64 + aarch64, CPython 3.12-3.14). Other platforms install from the sdist, which needs a Rust toolchain at build time.

Note: psycopg (the library) must be installed — it is a declared dependency. Django's stock postgresql backend, which django-vpg subclasses for schema/introspection/operations, imports it at load time. The IO path never uses it.

Quickstart

DATABASES = {
    "default": {
        "ENGINE": "django_vpg.backend",
        "NAME": "mydb",
        "USER": "postgres",
        "PASSWORD": "...",
        "HOST": "db.example.com",
        "PORT": "5432",
        # Optional driver OPTIONS (libpq-parity naming; psycopg3-compatible
        # pool dict):
        # "OPTIONS": {
        #     "pool": {"max_size": 20, "min_size": 2},   # default max_size: 10
        #     "sslmode": "verify-full",
        #     "sslrootcert": "/path/ca.pem",
        # },
    }
}

Supported OPTIONS keys: pool (psycopg3-compatible dict — max_size, min_size, timeout, max_lifetime, max_idle, name — or False for a dedicated single-slot connection per Django connection, e.g. under test runners), pool_size (shorthand for pool={"max_size": N}; default 10), prepare_threshold (psycopg3 semantics — int = prepare after N uses, 0 = prepare on first use, None = disable prepared statements; defaults to off, matching Django's postgresql backend, so opt in with an int), prepared_max, isolation_level, application_name, connect_timeout, keepalives, keepalives_idle, sslmode, sslrootcert, sslcert, sslkey, and server_settings. pgbouncer (bool) is accepted as a legacy alias for prepare_threshold=None; prefer the psycopg spelling, which — unlike pgbouncer — is a real psycopg key, so the same OPTIONS dict also works on the stock backend. Unknown keys warn once and are dropped.

The async story (three layers)

1. Sync ORM — stock Django, zero extra deps. Works everywhere Django works.
2. Async driver API — real asyncio, works anywhere. The raw driver (django_vpg._vpg_driver.RustPgDriver) exposes await-able query, execute, transaction, and COPY operations with no dependency on any async framework beyond asyncio itself; driver awaitables interoperate with gather, wait_for, shield, and cancellation. (django_vpg.dbapi itself is the sync DB-API 2.0 shim the Django engine builds on.)
3. Async ORM — via django-async-backend. Installing the django-vpg[async] extra activates AsyncDatabaseWrapper, which plugs into the async_connections framework. Without it, the backend cleanly degrades to sync-only (no ImportError).

Unifying Django's connections and async_connections (transaction management and isolation across both worlds) is active upstream work in django-async-backend that this package tracks closely.

Pooling and session semantics (psycopg is the reference)

Two rules, both deliberately psycopg's, because porting an app should not mean re-learning connection behaviour:

• The backend decides whether a transaction is open — never the driver. Every PostgreSQL response ends with a ReadyForQuery carrying the session's transaction state; libpq caches it as PQtransactionStatus and psycopg exposes it as Connection.info.transaction_status, which django-vpg does too. On release the pool applies psycopg_pool's policy: idle sends nothing, an open or failed block is rolled back, and a connection whose state cannot be established is discarded rather than reused. (Inferring this from SQL text instead is not sound — it misreads ROLLBACK TRANSACTION TO SAVEPOINT, COMMIT AND CHAIN, ABORT, comment-led SQL and PL/pgSQL bodies, and it cannot see statements that never went through the driver's own statement path.) A pool hook re-checks at checkout, so no return path can hand on a connection that is still inside a transaction.
• Session state is not reset between checkouts. Same as psycopg_pool, which ships no reset by default: GUCs set with SET, TEMP tables, session advisory locks and LISTEN registrations survive into whoever checks the connection out next. Reset them yourself if your application needs a clean session — and note DISCARD ALL is the wrong tool, because it deallocates prepared statements the driver's per-connection cache still refers to.

One more difference worth knowing if you use the async ORM: in async autocommit each statement borrows a pooled connection and returns it, so consecutive autocommit statements may run on different backend sessions — pg_backend_pid() is not stable, and TEMP tables, SET GUCs and advisory locks do not persist between them. The sync path pins one session per connection and matches psycopg. Transactions pin in both. This is a deliberate trade for throughput (holding no connection between statements lets a small pool serve many concurrent tasks), and it is safe rather than merely fast because of the status checks above. It will become a configurable choice before 1.0.

The tokio runtime

The driver runs on a fork-safe shared tokio runtime owned by the package (1 worker thread by default; VPG_TOKIO_WORKER_THREADS overrides). Prefork servers are safe: the runtime is rebuilt per child on first use.

Rust embedders get the same hook django-vcache offers: link vpg-pyo3 as an rlib, re-export its pyclasses into your own pymodule, and install a process-wide runtime with vpg_core::set_runtime_provider(my_get_runtime) so one tokio pool serves every driver in the process. (This is how GlitchTip's own Rust extension embeds the driver.) The hook returns false if a runtime had already been built — assert on it at module init to catch wiring mistakes.

Known limitations

• USE_TZ = True only (Django's default). timestamptz values are returned timezone-aware in UTC. The stock psycopg backend returns naive local datetimes when USE_TZ = False; django-vpg does not implement that conversion yet, so a USE_TZ = False project will hit naive/aware comparison errors on DateTimeFields.
• One statement per execute(). Multi-statement strings (cursor.execute("SET x = 1; SELECT 1")) raise a syntax error where psycopg3 runs them; the extended query protocol is one statement per Parse. Django's schema editor and test-runner paths batch through a dedicated internal path and are unaffected — this only bites ported raw-SQL code, which should split its statements.

Known naming wart: the vendored tokio-postgres buffer-cap patch reads the GT_PG_BUF_CAP_BYTES env var (512 KiB default retained-buffer cap); it will be renamed with a compatibility window before 1.0.

Development

./scripts/check.sh        # fmt + clippy -D warnings + cargo test + ruff
                          # (includes tests/wire.rs — an in-process scripted
                          #  PG wire server; no live postgres needed)

# Python suite (needs maturin + a live postgres for the marked tests):
python -m venv .venv && source .venv/bin/activate
pip install maturin pytest pytest-asyncio django psycopg django-async-backend
maturin develop --release
VPG_TEST_DSN=postgres://postgres@localhost:5432/postgres python -m pytest tests/

The workspace is two crates: crates/vpg-core (pure Rust, no pyo3, cargo test-able) and crates/vpg-pyo3 (thin bindings, cdylib + rlib). vendor/tokio-postgres carries a minimal retained-buffer-capacity patch — see vendor/tokio-postgres/GT_PATCH.md for rationale and the re-audit procedure on version bumps.

License

MIT