Skip to main content

Dump and load redis databases

Project description

https://api.travis-ci.org/p/redis-dump-load.png

Dumps Redis data sets into a format suitable for long-term storage (currently JSON) and loads data from such dump files back into Redis.

redis-dump was there first, but it is written in Ruby.

The output format is intended to be compatible with redis-dump.

Usage

redis-dump-load may be used as a module and also as a command line tool.

Module Usage

redis-dump-load exports a pickle-like interface, namely load, loads, dump and dumps functions. For example:

import redisdl

json_text = redisdl.dumps()

with open('path/to/dump.json', 'w') as f:
    # streams data
    redisdl.dump(f)

json_text = '...'
redisdl.loads(json_text)

with open('path/to/dump.json') as f:
    # streams data if ijson is installed
    redisdl.load(f)

See the streaming section below for more information about streaming.

Dump and load methods accept options as keyword arguments:

json_text = redisdl.dumps(encoding='iso-8859-1', pretty=True)

The arguments should always be passed in as keywords, i.e, do not rely on the order in which the parameters are currently listed. Options take string values unless otherwise noted. The options are as follows:

  • host: host name or IP address for redis server
  • port: port number for redis server
  • unix_socket_path: connect to redis via a Unix socket instead of TCP/IP; specify the path to the socket
  • password: specify password to connect to redis
  • db (integer): redis database to connect to
  • encoding: encoding to use for encoding or decoding the data, see Unicode section below
  • pretty (boolean, dump only): produce a pretty-printed JSON which is easier to read; currently this makes dump load entire data set into memory rather than stream it
  • empty (boolean, load only): empty the redis data set before loading the data
  • streaming_backend (string): ijson backend to use when loading via load method, if ijson is installed and streaming is thus used

Command Line Usage

redisdl.py can be used as a command line tool as follows:

# dump database 0
./redisdl.py > dump.json
./redisdl.py -o dump.json

# load into database 0
./redisdl.py -l < dump.json
./redisdl.py -l dump.json

For convenience, redisdl.py can be hard or soft linked as follows:

ln redisdl.py redis-dump
ln redisdl.py redis-load

Now it can be used thusly:

# dump database 0
./redis-dump > dump.json
./redis-dump -o dump.json

# load into database 0
./redis-load < dump.json
./redis-load dump.json

Symlinks work as well. “load” in the executable name triggers the loading mode, “dump” triggers the dumping mode, otherwise the default is to dump and -l option switches into the loading mode.

All options supported by the module API are accepted when redisdl is invoked as a command line tool. The command line options are:

  • -h/--help: help text
  • -H HOST/--host HOST: specify redis host
  • -p PORT/--port PORT: specify redis port
  • -s SOCKET_PATH/--socket SOCKET_PATH: connect to Unix socket at the specified path
  • -w PASSWORD/--password PASSWORD: password to use when connecting to redis
  • -d DATABASE/--db DATABASE: redis database to connect to (integer)
  • -E ENCODING/-encoding ENCODING: specify encoding to use
  • -o PATH/--output PATH: write dump to PATH rather than standard output
  • -y/--pretty (dumping only): pretty-print JSON
  • -e/--empty (loading only): empty redis data set before loading
  • -B BACKEND/--backend BACKEND (loading only): ijson streaming backend to use

Streaming

dump will stream data unless pretty is given and True.

load will stream data if ijson is installed. To determine whether redis-dump-load supports streaming data load, examine redisdl.have_streaming_load variable.

Default ijson streaming backend is python and ijson does not autoselect backends based on installed json libraries. To use a non-default ijson backend, either pass the desired backend as follows:

redisdl.load(io, streaming_backend='yajl2')

… or set the desired backend globally as follows:

redisdl.streaming_backend = 'yajl2'

Note: yajl2 backend fails when it is given strings on Python 3. Please open the files in binary mode and use BytesIO rather than StringIO objects.

Note: streaming loading is substantially slower than lump loading.

Unicode

Redis operates on bytes and has no concept of Unicode or encodings. JSON operates on Unicode strings and cannot serialize binary data. Therefore, redis-dump-load has to encode Unicode strings into byte strings when loading data into Redis and decode byte strings into Unicode strings when dumping data from Redis. By default redis-dump-load uses utf-8 for encoding and decoding. This behavior matches redis-py, whose default encoding is utf-8. A different encoding can be specified.

Concurrent Modifications

redis-dump-load does not lock the entire data set it is dumping, because Redis does not provide a way to do so. As a result, modifications to the data set made while a dump is in progress affect the contents of the dump.

Dependencies

Tests

redis-dump-load has a test suite. To run it, install nose and run:

nosetests

There are several tests that check for race conditions and as such take a long time to run. To skip them, invoke nose thusly:

nosetests -a '!slow'

License

Released under the 2 clause BSD license.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for redis-dump-load, version 0.4.0
Filename, size File type Python version Upload date Hashes
Filename, size redis-dump-load-0.4.0.tar.gz (7.4 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page