Skip to main content

Get started with sharded databases using ShardedModel, related fields, querysets, managers and a shard-aware database router.

Project description

Django Sharded adds ShardedModel, related fields, querysets, managers and a shard-aware database router to quickly get started with sharded databases.

You can start with one shard and add more shards later. A maximum of 255 shards are supported.

sharded.db.models is a drop-in replacement for django.db.models to help avoid import clutter.

At the moment, this package has been designed and tested to work only with PostgreSQL.

Quick start

  1. Add sharded as the first entry to your INSTALLED_APPS setting:

  2. Run python migrate to add sharded.Shard model to default database

  3. [OPTIONAL] Set a value for SHARDED_DB_PREFIX. Default prefix is shard_:

    SHARDED_DB_PREFIX = 'shard_'
  4. Add sharded.db.routers.ShardedRouter to DATABASE_ROUTERS setting:

    DATABASE_ROUTERS = ['sharded.db.routers.ShardedRouter',]
  5. Setup all available shards:

    #NOTE: Django Sharded accesses each shard using the name format '%s%03d' % (SHARDED_DB_PREFIX, n)
    DATABASES['shard_001'] = {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'shard',
        'USER': 'prjdbuser',
        'PASSWORD': 'supersecretpassword',
        'HOST': '',
        'PORT': '5432',
    DATABASES['shard_002'] = {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'shard_002',
        'USER': 'prjdbuser',
        'PASSWORD': 'supersecretpassword',
        'HOST': '',
        'PORT': '5432',
    DATABASES['shard_003'] = {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'shard_003',
        'USER': 'prjdbuser',
        'PASSWORD': 'supersecretpassword',
        'HOST': '',
        'PORT': '5432',
    DATABASES['shard_004'] = {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'shard',
        'USER': 'prjdbuser',
        'PASSWORD': 'supersecretpassword',
        'HOST': '',
        'PORT': '5432',
    #NOTE: If you use 'dj-database-url' or 'django-connection-url' (shameless self-plug),
    #      you can simplify the above using env vars like DATABASE_SHARD_nnn_URL
    import connection_url
    for shard in xrange(1,256):
        shard = '%s%03d' % ('shard_', shard) #TODO: Use SHARDED_DB_PREFIX if you've customized it above
        shard_env = 'DATABASE_' + shard.upper() + '_URL'
        if shard_env not in os.environ:
        DATABASES.setdefault(shard, connection_url.config(shard_env))
  6. Run python initshard <1..255> with optional parameters to initialize the shard

  7. Run python updateshard <1..255> –capacity <in_bytes> to keep track of available capacity in the shard

  8. Use ShardedModel as base for models that require sharding. Related models will automatically be included in the same shard and the ForeignKey field will also automatically use a big integer column:

    from sharded.db import models
    from sharded.models import Shard
    class HelloManager(models.ShardedManager):
        use_for_related_fields = True
        def create(self, **kwargs):
            if not self._db:
                self._db = str(Shard.objects.most_free_shard())
            return super(HelloManager, self).create(**kwargs)
    class Hello(models.ShardedModel):
        a_random_field = models.IntegerField()
        objects = HelloManager()
    class Foo(models.Model):
        hello = models.ForeignKey(Hello)
    class Bar(models.Model):
        hello = models.OneToOneField(Hello)
    class Baz(models.ShardedModel):
        hello = models.ForeignKey(Hello)
    class Herp(models.Model): #Unsharded model
        derp = models.CharField(max_length=8)
  9. Run python makemigrations and then, python migrate –all to apply migrations across all shards

  10. Add a cronjob to run python updateshard to update shard usage levels at regular intervals

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release. See tutorial on generating distribution archives.

Built Distribution

django_sharded-0.1.2-py2-none-any.whl (21.5 kB view hashes)

Uploaded 2 7

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page