Skip to main content

Add a DB connection pool using gevent to django

Project description

django-db-geventpool

CI

pypi version

pypi license

Another DB pool using gevent for PostgreSQL DB.

psycopg3

Django, since 4.2, supports psycopg3. One of the advantages is that gevent is supported without needing extra patches, just install the package

$ pip install psycopg[binary]

psycopg2

If gevent is not installed, the pool will use eventlet as fallback.

  • psycopg2>=2.5.1 for CPython 2 and 3 (or psycopg2-binary---see notes in the psycopg2 2.7.4 release)

  • psycopg2cffi>=2.7 for PyPy

    Patch psycopg2

    Before using the pool, psycopg2 must be patched with psycogreen, if you are using gunicorn webserver, a good place is the post_fork() function at the config file:

    from psycogreen.gevent import patch_psycopg     # use this if you use gevent workers
    from psycogreen.eventlet import patch_psycopg   # use this if you use eventlet workers
    
    def post_fork(server, worker):
        patch_psycopg()
        worker.log.info("Made Psycopg2 Green")
    

Settings

  • Set ENGINE in your database settings to:

      - For psycopg3: 'django_db_geventpool.backends.postgresql_psycopg3'
      - For psycopg2: 'django_db_geventpool.backends.postgresql_psycopg2'
      - For postgis: 'django_db_geventpool.backends.postgis'
    
  • Add MAX_CONNS to OPTIONS to set the maximun number of connections allowed to database (default=4)

  • Add REUSE_CONNS to OPTIONS to indicate how many of the MAX_CONNS should be reused by new requests. Will fallback to the same value as MAX_CONNS if not defined

  • Add 'CONN_MAX_AGE': 0 to settings to disable default django persistent connection feature. And read below note if you are manually spawning greenlets

For Django < 5.1:

DATABASES = {
    'default': {
        'ENGINE': 'django_db_geventpool.backends.postgresql_psycopg',
        'NAME': 'db',
        'USER': 'postgres',
        'PASSWORD': 'postgres',
        'HOST': '',
        'PORT': '',
        'ATOMIC_REQUESTS': False,
        'CONN_MAX_AGE': 0,
        'OPTIONS': {
            'MAX_CONNS': 20,
            'REUSE_CONNS': 10
        }
    }
}

For Django >= 5.1, native pool support should be disabled:

DATABASES = {
    'default': {
        'ENGINE': 'django_db_geventpool.backends.postgresql_psycopg',
        'NAME': 'db',
        'USER': 'postgres',
        'PASSWORD': 'postgres',
        'HOST': '',
        'PORT': '',
        'ATOMIC_REQUESTS': False,
        'CONN_MAX_AGE': 0,
        'OPTIONS': {
            'MAX_CONNS': 20,
            'REUSE_CONNS': 10.
            'pool': False,
        }
    }
}

Using ORM when not serving requests

If you are using django with celery (or other), or have code that manually spawn greenlets it will not be sufficient to set CONN_MAX_AGE to 0. Django only checks for long-live connections when finishing a request - So if you manually spawn a greenlet (or task spawning one) its connections will not get cleaned up and will live until timeout. In production this can cause quite some open connections and while developing it can hamper your tests cases.

To solve it make sure that each greenlet function (or task) either sends the django.core.signals.request_finished signal or calls django.db.close_old_connections() right before it ends

The decorator method with your function is preferred, but the other alternatives are also valid

from django_db_geventpool.utils import close_connection

@close_connection
def foo_func()
     ...

or

from django.core.signals import request_finished

def foo_func():
   ...
   request_finished.send(sender="greenlet")

or

from django.db import close_old_connections

def foo_func():
   ...
   close_old_connections()

Other pools

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

django_db_geventpool-4.0.8.tar.gz (9.2 kB view details)

Uploaded Source

Built Distribution

django_db_geventpool-4.0.8-py2.py3-none-any.whl (13.4 kB view details)

Uploaded Python 2Python 3

File details

Details for the file django_db_geventpool-4.0.8.tar.gz.

File metadata

  • Download URL: django_db_geventpool-4.0.8.tar.gz
  • Upload date:
  • Size: 9.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for django_db_geventpool-4.0.8.tar.gz
Algorithm Hash digest
SHA256 8ea100e481515272f76c77f4df3bb3401ce838c97549782d0d30faf990d0cbf4
MD5 2cb701cd914eb12bf5f933ca574acc4d
BLAKE2b-256 000f0eaf58738605476b4c1cfad5ae2ed0d062572a73383b204c910e6d92be6c

See more details on using hashes here.

File details

Details for the file django_db_geventpool-4.0.8-py2.py3-none-any.whl.

File metadata

File hashes

Hashes for django_db_geventpool-4.0.8-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 80f43e1eb05e137ce0672ac5a29b859bbca4f3ee1ea00d1ea521109ee9a68d06
MD5 19a4cf340b26228f1da387107ddc69cf
BLAKE2b-256 0dd0488c64fa100b2194cc3c8d36c7a92c1e1268149c912ca35c3b34136420cd

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page