django-project-backup

Django project backup application.

(This is alpha software and is under heavy development)

Setup

Add the following lines to settings.py

# django
INSTALLED_APPS += [
    'django_project_backup'
]

# django-project-backup
SERIALIZATION_MODULES = {
    'dpb_couchdb_datastore': 'django_project_backup.utils.couchdb.serializers'
}

DJANGO_PROJECT_BACKUP_MODE = env('DJANGO_PROJECT_BACKUP_MODE', default='incremental')

DJANGO_PROJECT_BACKUP_PROJECT_NAME = env('DJANGO_PROJECT_BACKUP_PROJECT_NAME', default='django_project_backup')
DJANGO_PROJECT_BACKUP_SHARD_NAME = env('DJANGO_PROJECT_BACKUP_SHARD_NAME', default='1')

# realtime
DJANGO_PROJECT_BACKUP_REALTIME = env.bool('DJANGO_PROJECT_BACKUP_REALTIME', default=True)
DJANGO_PROJECT_BACKUP_DO_REALTIME_COUCHDB_BACKUP = env.bool('DJANGO_PROJECT_BACKUP_DO_REALTIME_COUCHDB_BACKUP',
                                                            default=True)
DJANGO_PROJECT_BACKUP_DO_REALTIME_LOG_BACKUP = env.bool('DJANGO_PROJECT_BACKUP_DO_REALTIME_LOG_BACKUP',
                                                        default=True)

DJANGO_PROJECT_BACKUP_EXCLUDED_MODELS = env.list('DJANGO_PROJECT_BACKUP_EXCLUDED_MODELS', default=[
    'sessions.session',
    'admin.logentry',
    'django_sso_app.passepartout',
    'django_sso_app.device',
    'easy_thumbnails.thumbnail'
])

DJANGO_PROJECT_BACKUP_PUBLIC_ASSETS_FOLDERS = env.list('DJANGO_PROJECT_BACKUP_PUBLIC_ASSETS_FOLDERS',
                                                        default=[str(PUBLIC_ROOT)])

DJANGO_PROJECT_BACKUP_PRIVATE_ASSETS_FOLDERS = env.list('DJANGO_PROJECT_BACKUP_PRIVATE_ASSETS_FOLDERS',
                                                        default=[str(PRIVATE_ROOT)])

DJANGO_PROJECT_BACKUP_DESTINATION_FOLDER = env('DJANGO_PROJECT_BACKUP_DESTINATION_FOLDER',
                                               default=os.path.join(ROOT_DIR, 'backups'))

DJANGO_PROJECT_BACKUP_COUCHDB_DATASTORE_URL = env('DJANGO_PROJECT_BACKUP_COUCHDB_DATASTORE_URL', default='http://127.0.0.1:5984')
DJANGO_PROJECT_BACKUP_COUCHDB_DATASTORE_USER = env('DJANGO_PROJECT_BACKUP_COUCHDB_DATASTORE_USER', default='admin')
DJANGO_PROJECT_BACKUP_COUCHDB_DATASTORE_PASSWORD = env('DJANGO_PROJECT_BACKUP_COUCHDB_DATASTORE_PASSWORD', default='couchdb')
# couchdb db index
DJANGO_PROJECT_BACKUP_COUCHDB_DATASTORE_DATABASE_NAME = env('DJANGO_PROJECT_BACKUP_COUCHDB_DATASTORE_DATABASE_NAME',
                                                            default='django_project_backup')

DJANGO_PROJECT_BACKUP_DO_FAILSAFE_BACKUP = env.bool('DJANGO_PROJECT_BACKUP_DO_FAILSAFE_BACKUP',
                                                    default=True)
DJANGO_PROJECT_BACKUP_FAILSAFE_BACKUP_PATH = env('DJANGO_PROJECT_BACKUP_FAILSAFE_BACKUP_PATH',
                                                 default=os.path.join(DJANGO_PROJECT_BACKUP_DESTINATION_FOLDER, 'failed'))

# log realtime
LOGGING = {
    ...

    "formatters": {
        "dpb_serialized_model": {
            "format": "%(message)s"
        }
    },
    'handlers': {
        "dpb_realtime_log_backup_file": {
            "level": "INFO",
            "class": "logging.handlers.RotatingFileHandler",
            "filename": os.path.join(ROOT_DIR, "logs", "django_project_backup.realtime.log"),
            "maxBytes": 1024 * 1024 * 4000,  # 4GB
            "backupCount": 10,
            "formatter": "dpb_serialized_model",
        },
    },
    'loggers': {
        ...

        'django_project_backup.backup_logger': {
            'handlers': ['dpb_realtime_log_backup_file'],
            'level': 'INFO',
            'propagate': False,
        },
    },
}

# django-filer
# store files as payload
FILER_DUMP_PAYLOAD = True

Usage

Filesystem

• Backup DB

  $ python manage.py dpb_filesystem_dumpdata
  

• Backup assets

  $ python manage.py dpb_filesystem_assets_backup
  

Couchdb

• Backup all

  $ python manage.py dpb_couchdb_dumpdata
  

• Restore all

  $ python manage.py dpb_couchdb_loaddata
  

Sandbox

Docker

Dependencies

• Docker
• Docker Compose

Installation

Run the following commands:

git clone https://bitbucket.org/pai/django-project-backup.git
cd django-project-backup
# build containers
docker-compose up --build -d
# start containers
docker-compose up
# load initial data
docker-compose run app /venv/bin/python manage.py load_initial_data
# perform first backup
docker-compose run app /venv/bin/python manage.py dbp_couchdb_dumpdata

The demo site will now be accessible at http://localhost:8000/ and the django admin interface at http://localhost:8000/admin/.

Log into the admin with the credentials admin / admin.

Important: This docker-compose.yml is configured for local testing only, and is not intended for production use.

Debugging

To tail the logs from the Docker containers in realtime, run:

docker-compose logs -f

Setup with Virtualenv

You can run the demo locally without setting up Docker and simply use Virtualenv, which is the recommended installation approach for Django itself.

Dependencies

• Python 3.6, 3.7 or 3.8
• Virtualenv
• VirtualenvWrapper (optional)

Installation

With PIP and virtualenvwrapper installed, run:

mkvirtualenv django_project_backup
python --version

Confirm that this is showing a compatible version of Python 3.x. If not, and you have multiple versions of Python installed on your system, you may need to specify the appropriate version when creating the virtualenv:

deactivate
rmvirtualenv django_project_backup
mkvirtualenv django_project_backup --python=python3.7
python --version

Now we're ready to set up the sandbox demo project itself:

cd ~/dev [or your preferred dev directory]
git clone https://bitbucket.org/pai/django-project-backup.git
cd django_project_backup/
pip install -r requirements/base.txt

Next, we'll set up our local environment variables. We use django-dotenv to help with this. It reads environment variables located in a file name .env in the top level directory of the project. The only variable we need to start is DJANGO_SETTINGS_MODULE:

$ cp backend/settings/local.py.example backend/settings/local.py
$ echo "DJANGO_SETTINGS_MODULE=backend.settings.local" > .env

To set up your database and load initial data, run the following commands:

./manage.py migrate
./manage.py load_initial_data
./manage.py runserver

Log into the admin with the credentials admin / admin.

Known issues

Since django-project-backup relies on natural primary keys when in realtime mode if one of those fields is updated there is no efficient way to update related objects keys.

Note

This project has been set up using PyScaffold 3.2.3. For details and usage information on PyScaffold see https://pyscaffold.org/.