Distributed databases setup in Django
Project description
django-distributed
A multi database setup for Django.
Tested with www.lodlaw.com, our setup consists of:
- one write database based in Australia corresponding to an Australian cluser
- one read database based in Europe corresponding to an European cluster
Installing
pip install django-distributed
Requirements
Required configs
In settings.py
of Django, you need to define the following variables for the follower configuration:
Variable | Type | Description |
---|---|---|
LEADER_DATABASE |
str | The name of the leader database (i.e. default ) |
FOLLOWER_DATABASES |
list[str] | A list of follower databases (i.e. ['replica1', 'replica2'] ) |
REPLICATION_LAG |
number | The replication lag in seconds between the read and write instances. |
Database router
In addition, you need to use our custom DistributedRouter
database router. In settings.py
, add in:
DATABASE_ROUTERS = ('django_distributed.SessionCachedRouter',)
MIDDLEWARE = ('crum.CurrentRequestUserMiddleware',) + MIDDLEWARE
Customize database router
Our database router is session-based. It means that we decide to which database to read from or write to based on the user session.
However, if you want to use another session back end, you have two options with our setup:
- Extend the class
CachedRouter
and implement the methodget_cache_key
. Define your custom cache key there; or - Extend the class
DistributedRouter
and implement the methodis_recently_updated
. Our distributed router logic will read from the write database if it is recently updated else replica database.
Optional configs
If you want to use our own session engine, you can do so by specifying in settings.py
SESSION_ENGINE = 'django_distributed.session_engine'
Session engine explanation
This session engine is using both cache store and database store.
Because replication is not fast enough, a session written in the leader database will not appear in the replicas right away. Hence, the user might get kicked out after logging in, as the session is not yet replicated. Of course, we could do R/W all on the master database, but it would be slow, since our R and W databases are located far away from each other.
Therefore, we have created this session engine, it will first add the session to the cache first, in our case, memcached. Then it will update the database. Same for reading session, cache store goes first then database store. Nevertheless, what is different here from the Django cached_db
is the write operation, it is offloaded to another thread.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.