Dump and load redis databases
Dumps Redis data sets into a format suitable for long-term storage (currently JSON) and loads data from such dump files back into Redis.
redis-dump was there first, but it is written in Ruby.
The output format is intended to be compatible with redis-dump.
redis-dump-load may be used as a module and also as a command line tool.
redis-dump-load exports a pickle-like interface, namely load, loads, dump and dumps functions. For example:
import redisdl json_text = redisdl.dumps() with open('path/to/dump.json', 'w') as f: # streams data redisdl.dump(f) json_text = '...' redisdl.loads(json_text) with open('path/to/dump.json') as f: # streams data if ijson is installed redisdl.load(f)
See the streaming section below for more information about streaming.
Dump and load methods accept options as keyword arguments:
json_text = redisdl.dumps(encoding='iso-8859-1', pretty=True)
The arguments should always be passed in as keywords, i.e, do not rely on the order in which the parameters are currently listed. Options take string values unless otherwise noted. The options are as follows:
- host: host name or IP address for redis server
- port: port number for redis server
- unix_socket_path: connect to redis via a Unix socket instead of TCP/IP; specify the path to the socket
- password: specify password to connect to redis
- db (integer): redis database to connect to
- encoding: encoding to use for encoding or decoding the data, see Unicode section below
- pretty (boolean, dump only): produce a pretty-printed JSON which is easier to read; currently this makes dump load entire data set into memory rather than stream it
- empty (boolean, load only): empty the redis data set before loading the data
- streaming_backend (string): ijson backend to use when loading via load method, if ijson is installed and streaming is thus used
Command Line Usage
redisdl.py can be used as a command line tool as follows:
# dump database 0 ./redisdl.py > dump.json ./redisdl.py -o dump.json # load into database 0 ./redisdl.py -l < dump.json ./redisdl.py -l dump.json
For convenience, redisdl.py can be hard or soft linked as follows:
ln redisdl.py redis-dump ln redisdl.py redis-load
Now it can be used thusly:
# dump database 0 ./redis-dump > dump.json ./redis-dump -o dump.json # load into database 0 ./redis-load < dump.json ./redis-load dump.json
Symlinks work as well. “load” in the executable name triggers the loading mode, “dump” triggers the dumping mode, otherwise the default is to dump and -l option switches into the loading mode.
All options supported by the module API are accepted when redisdl is invoked as a command line tool. The command line options are:
- -h/--help: help text
- -H HOST/--host HOST: specify redis host
- -p PORT/--port PORT: specify redis port
- -s SOCKET_PATH/--socket SOCKET_PATH: connect to Unix socket at the specified path
- -w PASSWORD/--password PASSWORD: password to use when connecting to redis
- -d DATABASE/--db DATABASE: redis database to connect to (integer)
- -E ENCODING/-encoding ENCODING: specify encoding to use
- -o PATH/--output PATH: write dump to PATH rather than standard output
- -y/--pretty (dumping only): pretty-print JSON
- -e/--empty (loading only): empty redis data set before loading
- -B BACKEND/--backend BACKEND (loading only): ijson streaming backend to use
dump will stream data unless pretty is given and True.
load will stream data if ijson is installed. To determine whether redis-dump-load supports streaming data load, examine redisdl.have_streaming_load variable.
Default ijson streaming backend is python and ijson does not autoselect backends based on installed json libraries. To use a non-default ijson backend, either pass the desired backend as follows:
… or set the desired backend globally as follows:
redisdl.streaming_backend = 'yajl2'
Note: yajl2 backend fails when it is given strings on Python 3. Please open the files in binary mode and use BytesIO rather than StringIO objects.
Note: streaming loading is substantially slower than lump loading.
Redis operates on bytes and has no concept of Unicode or encodings. JSON operates on Unicode strings and cannot serialize binary data. Therefore, redis-dump-load has to encode Unicode strings into byte strings when loading data into Redis and decode byte strings into Unicode strings when dumping data from Redis. By default redis-dump-load uses utf-8 for encoding and decoding. This behavior matches redis-py, whose default encoding is utf-8. A different encoding can be specified.
redis-dump-load does not lock the entire data set it is dumping, because Redis does not provide a way to do so. As a result, modifications to the data set made while a dump is in progress affect the contents of the dump.
redis-dump-load has a test suite. To run it, install nose and run:
There are several tests that check for race conditions and as such take a long time to run. To skip them, invoke nose thusly:
nosetests -a '!slow'
Released under the 2 clause BSD license.
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size redis-dump-load-0.4.0.tar.gz (7.4 kB)||File type Source||Python version None||Upload date||Hashes View|