Skip to main content

Consistent partial database dump utility

Project description

XDump

Build Status Coverage Status Documentation Status Latest PyPI version

XDump is an utility to make partial consistent dump and load it into the database.

The idea is to provide an ability to specify what to include in the dump via SQL queries.

Installation

XDump can be obtained with pip:

$ pip install xdump

Usage example

Making a dump (on production replica for example):

>>> from xdump.postgresql import PostgreSQLBackend
>>>
>>> backend = PostgreSQLBackend(dbname='app_db', user='prod', password='pass', host='127.0.0.1', port='5432')
>>> backend.dump(
    '/path/to/dump.zip',
    full_tables=['groups'],
    partial_tables={'employees': 'SELECT * FROM employees ORDER BY id DESC LIMIT 2'}
)

Load a dump on you local machine:

>>> backend = PostgreSQLBackend(dbname='app_db', user='local', password='pass', host='127.0.0.1', port='5432')
>>> backend.load('/path/to/dump.zip')

RDBMS support

At the moment only the following are supported:

  • PostgreSQL
  • SQLite

Django support

Add xdump.extra.django to your INSTALLED_APPS settings:

INSTALLED_APPS = [
   ...,
   'xdump.extra.django',
]

Add XDUMP to your project settings file. It should contain minimum 2 entries:

  • FULL_TABLES - a list of tables, that should be fully dumped.
  • PARTIAL_TABLES - a dictionary with table_name: select SQL
XDUMP = {
    'FULL_TABLES': ['groups'],
    'PARTIAL_TABLES': {'employees': 'SELECT * FROM employees WHERE id > 100'}
}

Optionally you could use a custom backend:

XDUMP = {
    ...,
    'BACKEND': 'importable.string',
}

Run xdump command:

$ ./manage.py xdump dump.zip

Run xload command:

$ ./manage.py xload dump.zip

Possible options to both commands:

  • alias - allows you to choose database config from DATABASES, that is used during the execution;
  • backend - importable string, that leads to custom dump backend class.

The following make command could be useful to get a configured dump from production to your local machine:

sync-production:
    ssh -t $(TARGET) "DJANGO_SETTINGS_MODULE=settings.production /path/to/manage.py xdump /tmp/dump.zip"
    scp $(TARGET):/tmp/dump.zip ./dump.zip
    ssh -t $(TARGET) "rm /tmp/dump.zip"
    DJANGO_SETTINGS_MODULE=settings.local $(PYTHON) manage.py xload ./dump.zip

And usage is:

$ make sync-production TARGET=john@production.com PYTHON=/path/to/python/in/venv

Python support

XDump supports Python 3.4, 3.5 & 3.6.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for xdump, version 0.3.1
Filename, size File type Python version Upload date Hashes
Filename, size xdump-0.3.1.tar.gz (11.5 kB) File type Source Python version None Upload date Hashes View
Filename, size xdump-0.3.1-py2.py3-none-any.whl (12.2 kB) File type Wheel Python version py2.py3 Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page