django-postgres-hot-upgrade 1.1.0

Django app that resets django.contrib.postgres caches that store postgres extensions OIDs

django-postgres-hot-upgrade

Let Django clear its PostgreSQL extensions OIDs cache, making it possible to update the PostgreSQL version to a new major version on the fly.

The how

$ pip install django-postgres-hot-upgrade

INSTALLED_APPS = [
    ...,
    # Warning: django_postgres_hot_upgrade requires to be placed before
    # django.contrib.postgres otherwise it will not work.
    'django_postgres_hot_upgrade',
    'django.contrib.postgres',
    ...,
]

The why

PostgreSQL keeps internal ids of for various objects (OIDs). This includes loaded extentions. In order to interact with those extensions, Django needs to know these IODs, so it loads them and, in order to avoid unneeded requests, it caches them in memory for the duration of the process.

Several PostgreSQL servers running the same version of PostgreSQL will have consistent OIDs but when you upgrade, OIDs can change. If one uses a PostgreSQL load balancer such as pgbouncer or pgpool, one could be tempted to migrate between major PostgreSQL versions on the fly, avoinding downtime. Indeed, for sufficiently recent versions of PostgreSQL, this would work, apart from the OID problem: if OIDs change, Django needs to update its cache.

django_postgres_hot_upgrade memorizes the postgres version of the server after each connection. When the version is updated, it clears the internal Django OIDs cache, forcing Django to fetch the new values.

The rest

Compatibility: Please refer to versions of Python and Django tested in tox.ini.

License: MIT

Code of Conduct: This project is placed under the Contributor Coveneant. Please report any abuse to joachim.jablon at people-doc.com.

[Maintainers] The ugly part

Apart from its unit tests, this package has an integration test. In order to test the feature, we need to simulate a change of OIDs caused by a live update from PG10 to PG12 in a controlled CI environment. This is the most fragile part of the lib, and the most likely to break in the future. Here's what you need to know:

• docker-compose.yml define two databases postgres10 and postgres12 listening on 5432 and 5433 respectively.
• tests/django_settings.py define a default database using libpq envvars. Note that in the settings, we requests the tests to run on the normal database instead of dedicated test_<foo> database.
• The OIDs are created by Postgres when installing the extensions. This happens in tests/migrations/0001_initial.py. The DJANGO_REVERSE_OPERATIONS env var controls the order of the 2 extensions creation. Running the PG10 migration in normal order and the PG12 migration in reverse order ensures the OIDs will be different.
• The runtests script ensure the migrations run on both databases in the decided order, then launches the test. Without this, the integration test would likely fail because the OIDs would be the same in the two databases.
• tox calls runtests.
• GitHub Actions call tox.

The following work to launch tests locally:

• run tox or runtests on fresh databases
• run pytests if you know the OIDs are already properly set on the 2 databases