Horizontal database groups and a metadata store

HORIZONTAL_CONFIG = {
    'GROUPS': {
        'group1': {  # The name of database horizontal partitioning group
            'DATABASES': {
                1: {
                    'write': 'member1-primary',
                    'read': ['member1-replica-1', 'member1-replica-2'],  # Pick randomly by router
                },
                2: {
                    'write': 'member2-primary',
                    'read': ['member2-replica'],
                },
                3: {
                    'write': 'a3',  # Used by 'read' too
                },
            },
            'PICKABLES': [2, 3],  # Group member keys to pick new database
        },
    },
    'METADATA_MODEL': 'app.HorizontalMetadata',  # Metadata store for horizontal partition key and there database
}

Database router

DATABASE_ROUTERS = (
    'horizon.routers.HorizontalRouter',
    ...
)

Metadata store

from horizon.models import AbstractHorizontalMetadata

class HorizontalMetadata(AbstractHorizontalMetadata):
    pass

In the example, metadata store keep user’s pk and that index of horizontal database (1, 2 or 3).

Sharded model

from django.conf import settings

from horizon.manager import HorizontalManager  # For Django<1.10
from horizon.models import AbstractHorizontalModel


class SomeLargeModel(AbstractHorizontalModel):
    user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE)
    ...

    objects = HorizontalManager()  # For Django<1.10

    class Meta(object):
        horizontal_group = 'group1'  # Group name
        horizontal_key = 'user'  # Group key

In many cases use UUIDField field for id. The AbstractHorizontalModel uses UUIDField as a them id field in default.

Using a model

from django.contrib.auth import get_user_model


user_model = get_user_model()
user = user_model.objects.get(pk=1)

# Get by foreign instance
SomeLargeModel.objects.filter(uses=user)

# Get by foreign id
SomeLargeModel.objects.filter(uses_id=user.id)

# django.db.utils.IntegrityError occured when not specify horizontal key field to filter
SomeLargeModel.objects.all()

Model limitations

• django.db.utils.IntegrityError occured when not specify horizontal key field to filter

  SomeLargeModel.objects.all()

• Cannot lookup by foreign key field, cause there are other (like default) database

  list(self.user.somelargemodel_set.all())