Add a DB connection pool using gevent to django
Project description
django-db-geventpool
Another DB pool using gevent for PostgreSQL DB.
psycopg3
Django, since 4.2, supports psycopg3. One of the advantages is that gevent is supported without needing extra patches, just install the package
$ pip install psycopg[binary]
psycopg2
If gevent is not installed, the pool will use eventlet as fallback.
-
psycopg2>=2.5.1
for CPython 2 and 3 (or psycopg2-binary---see notes in the psycopg2 2.7.4 release) -
psycopg2cffi>=2.7
for PyPyPatch psycopg2
Before using the pool, psycopg2 must be patched with psycogreen, if you are using gunicorn webserver, a good place is the post_fork() function at the config file:
from psycogreen.gevent import patch_psycopg # use this if you use gevent workers from psycogreen.eventlet import patch_psycopg # use this if you use eventlet workers def post_fork(server, worker): patch_psycopg() worker.log.info("Made Psycopg2 Green")
Settings
Set ENGINE in your database settings to:
- For psycopg3: 'django_db_geventpool.backends.postgresql_psycopg3' - For psycopg2: 'django_db_geventpool.backends.postgresql_psycopg2' - For postgis: 'django_db_geventpool.backends.postgis'
Add MAX_CONNS to OPTIONS to set the maximun number of connections allowed to database (default=4)
Add REUSE_CONNS to OPTIONS to indicate how many of the MAX_CONNS should be reused by new requests. Will fallback to the same value as MAX_CONNS if not defined
Add 'CONN_MAX_AGE': 0 to settings to disable default django persistent connection feature. And read below note if you are manually spawning greenlets
DATABASES = {
'default': {
'ENGINE': 'django_db_geventpool.backends.postgresql_psycopg',
'NAME': 'db',
'USER': 'postgres',
'PASSWORD': 'postgres',
'HOST': '',
'PORT': '',
'ATOMIC_REQUESTS': False,
'CONN_MAX_AGE': 0,
'OPTIONS': {
'MAX_CONNS': 20,
'REUSE_CONNS': 10
}
}
}
Using ORM when not serving requests
If you are using django with celery (or other), or have code that manually spawn greenlets it will not be sufficient to set CONN_MAX_AGE to 0. Django only checks for long-live connections when finishing a request - So if you manually spawn a greenlet (or task spawning one) its connections will not get cleaned up and will live until timeout. In production this can cause quite some open connections and while developing it can hamper your tests cases.
To solve it make sure that each greenlet function (or task) either sends the django.core.signals.request_finished signal or calls django.db.close_old_connections() right before it ends
The decorator method with your function is preferred, but the other alternatives are also valid
from django_db_geventpool.utils import close_connection
@close_connection
def foo_func()
...
or
from django.core.signals import request_finished
def foo_func():
...
request_finished.send(sender="greenlet")
or
from django.db import close_old_connections
def foo_func():
...
close_old_connections()
Other pools
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for django_db_geventpool-4.0.4.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 95791ba1557402e0c00b3dbbdaef710c7c369102d5d2d165905820afc4e5a7c7 |
|
MD5 | f08acbb52490195268cc4a8ff381df90 |
|
BLAKE2b-256 | a68cb0fde24cb5855f9918ae9dc2eb769b5e3fc570ec6cad2556b1eb8a0d9077 |
Hashes for django_db_geventpool-4.0.4-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f08c0c0821d299fbf54f4316e89954bc9d4e2ac260815b41e344e6eb28a206f5 |
|
MD5 | 109009f92812882db08d036d2314b12f |
|
BLAKE2b-256 | b6dd68a9080c32fa8930177e5318b1e2bc263e7392bf79ff5eff8e8c188e6f33 |