django-dmf (Django Data Migration Framework)

A Version Controlled, Fully Managed Data Migration Framework for Django that kicks in post Django's built-in Schema Migration.

django-dmf helps to keep Schema Migrations and Data Migrations isolated from each other. This ensures that if one changes, it doesn't impact the other, and both migrations can be independently managed, versioned, and executed.

Features

• Isolated from Schema Migrations: Keeps data manipulation out of Django's migrations files.
• Version Controlled: Each data migration task is versioned. The framework tracks executions in the database and ensures a specific version of a task is never run twice.
• Transactional by Default: Tasks run within an atomic database transaction. If a task fails, the transaction is rolled back safely.
• Retry Mechanism: Configurable retry policies for data migrations, handling transient database locks or network issues.
• Schema Aware: Native support for multi-tenant applications using PostgreSQL schemas (can specify if a task should run in the public schema, a custom schema, or both).

Installation

You can install django-dmf from PyPI using pip:

pip install django_dmf

Configuration

1. Add dm to your INSTALLED_APPS in settings.py:

INSTALLED_APPS = [
    # ...
    'dm',
    # ...
]

2. Run schema migrations to create the necessary database table (DataMigrationExecution) that keeps track of the data migration statuses:

python manage.py migrate dm

3. Configure your Data Migration Settings in settings.py:

# settings.py

# A list of string paths to your Data Migration Task classes
DATA_MIGRATION_REGISTRY = [
    'my_app.dm.tasks.FeatureSwitchDataMigrationTask',
]

# (Optional) Retry configuration
DATA_MIGRATION_TOTAL_RETRIES = 5             # Default is 5
DATA_MIGRATION_RETRY_DELAY_IN_SECONDS = 5    # Default is 5

Usage

1. Create a Data Migration Task

Create a new Python file for your task and inherit from BaseDataMigrationTask.

# my_app/dm/tasks.py

from dm.tasks.dm.base import BaseDataMigrationTask
from my_app.models import UserProfile

class FeatureSwitchDataMigrationTask(BaseDataMigrationTask):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        # Required: Set a unique version for this task.
        # The framework will use this to track execution status.
        self.version = 1

        # Optional: Multi-tenant schema settings
        # self.run_in_public_schema = True
        # self.run_in_custom_schema = True

    def _run(self):
        # Your data migration logic goes here.
        # This block is executed within an atomic database transaction.
        UserProfile.objects.filter(is_active=None).update(is_active=True)

2. Register the Task

Ensure your task is added to DATA_MIGRATION_REGISTRY in your settings.py.

DATA_MIGRATION_REGISTRY = [
    'my_app.dm.tasks.FeatureSwitchDataMigrationTask',
]

3. Run Data Migrations

Use the provided management command to execute pending data migrations:

python manage.py migrate_data

The command will iterate through your registry, check the execution history in the database, and run any tasks (or updated versions of tasks) that haven't been completed successfully.

How it Works

Under the hood, django-dmf creates a DataMigrationExecution record for every task and version it encounters.

• IN_PROGRESS: Marked when the task begins.
• COMPLETED: Marked when _run() completes without raising any exceptions.
• FAILED: Marked if an exception is raised inside _run(). The database transaction is rolled back, and the framework will optionally retry based on your DATA_MIGRATION_TOTAL_RETRIES setting.

License

This project is licensed under the MIT License - see the LICENSE file for details.