Skip to main content
This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).
Help us improve Python packaging - Donate today!

Consistent partial database dump utility

Project Description

XDump

XDump is an utility to make partial consistent dump and load it into the database.

The idea is to provide an ability to specify what to include in the dump via SQL queries. Responsibility for dump integrity lays on the user in this case.

Installation

XDump can be obtained with pip:

$ pip install xdump

Usage example

Making a dump (on production replica for example):

>>> EMPLOYEES_SQL = '''
WITH RECURSIVE employees_cte AS (
  SELECT *
  FROM recent_employees
  UNION
  SELECT E.*
  FROM employees E
  INNER JOIN employees_cte ON (employees_cte.manager_id = E.id)
), recent_employees AS (
  SELECT *
  FROM employees
  ORDER BY id DESC
  LIMIT 2
)
SELECT * FROM employees_cte
'''
>>> from xdump.postgresql import PostgreSQLBackend
>>> backend = PostgreSQLBackend(dbname='app_db', user='prod', password='pass', host='127.0.0.1', port='5432')
>>> backend.dump('/path/to/dump.zip', full_tables=['groups'], partial_tables={'employees': EMPLOYEES_SQL})

Load a dump on you local machine:

from xdump.postgresql import PostgreSQLBackend
>>> backend = PostgreSQLBackend(dbname='app_db', user='local', password='pass', host='127.0.0.1', port='5432')
>>> backend.load('/path/to/dump.zip')

RDBMS support

At the moment only the following are supported:

  • PostgreSQL
  • SQLite

Django support

Add xdump.extra.django to your INSTALLED_APPS settings:

INSTALLED_APPS = [
   ...,
   'xdump.extra.django',
]

Add XDUMP to your project settings file. It should contain minimum 2 entries:

  • FULL_TABLES - a list of tables, that should be fully dumped.
  • PARTIAL_TABLES - a dictionary with table_name: select SQL
XDUMP = {
    'FULL_TABLES': ['groups'],
    'PARTIAL_TABLES': {'employees': 'SELECT * FROM employees WHERE id > 100'}
}

Optionally you could use a custom backend:

XDUMP = {
    ...,
    'BACKEND': 'importable.string',
}

Run xdump command:

$ ./manage.py xdump dump.zip

Run xload command:

$ ./manage.py xload dump.zip

Possible options to both commands:

  • alias - allows you to choose database config from DATABASES, that is used during the execution;
  • backend - importable string, that leads to custom dump backend class.

The following make command could be useful to get a configured dump from production to your local machine:

sync-production:
    ssh -t $(TARGET) "DJANGO_SETTINGS_MODULE=settings.production /path/to/manage.py xdump /tmp/dump.zip"
    scp $(TARGET):/tmp/dump.zip ./dump.zip
    ssh -t $(TARGET) "rm /tmp/dump.zip"
    DJANGO_SETTINGS_MODULE=settings.local $(PYTHON) manage.py xload ./dump.zip

And usage is:

$ make sync-production TARGET=john@production.com PYTHON=/path/to/python/in/venv

Python support

XDump supports only Python 3.6 at the moment (but the supported versions list will be extended in the future).

Release History

This version
History Node

0.2.0

History Node

0.1.2

History Node

0.1.1

History Node

0.1

History Node

0.0.1

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, Size & Hash SHA256 Hash Help File Type Python Version Upload Date
xdump-0.2.0-py2.py3-none-any.whl
(13.0 kB) Copy SHA256 Hash SHA256
Wheel py2.py3 Jan 7, 2018
xdump-0.2.0.tar.gz
(10.0 kB) Copy SHA256 Hash SHA256
Source None Jan 7, 2018

Supported By

Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Google Google Cloud Servers DreamHost DreamHost Log Hosting