Mirror Redis Traffic to another redis node
Project description
Mirror Redis Traffic to another redis node
redis-mirror
Realtime Redis Traffic Mirror to another instance, this script reads the STDOUT from redis-cli monitor
command and mirrors the keys to another instance.
Use Case/Note
In some production/development cases, you need to mirror Redis traffic to another node in order to do some investigation or debugging.
- This script does not set
TTL
to mirrored key since you need it for debugging. - For now, this script supports
set
,hset
. I will add the other command soon, or if you have the time, please create a pull request :).
TO DO:
- Add the most common redis command to mirror script.
- Add TTL as an option in mirrored redis instance, plus add an option to expand the origin
TTL
. - Support cluster to a single redis instance or vice versa.
- Add more option such as
DB
Name,Host
,Port
, etc. - Improve mirrored value without any modification.
- Add option to dump all keys name to file for further analysis.
Option
Options:
--host TEXT Destination redis host/IP.
--port INTEGER Destination redis port.
--db INTEGER Destination redis DB.
--auth TEXT Destination redis auth info.
--counter INTEGER number of keys to mirror.
--help Show this message and exit.
Useages
redis-cli monitor | python redismirror.py
Exmaple output:
$redis-cli monitor | redismirror run
Init connection redis..
Skip line | OK
Key mirroed sucessfully | 05439891-2f59-45c3-931d-4918d3aaf121_1_uuid_ran
Key mirroed sucessfully | 05439891-2f59-45c3-931d-4918d3aaf121_1_date_ran
Key mirroed sucessfully | 05439891-2f59-45c3-931d-4918d3aaf121_1_date2_ran
Key mirroed sucessfully | 05439891-2f59-45c3-931d-4918d3aaf121_1_json_ran
Key mirroed sucessfully | 05439891-2f59-45c3-931d-4918d3aaf121_1_image_ran
Key mirroed sucessfully | 1609021903_1_uuid_ran
Key mirroed sucessfully | 1609021903_1_date_ran
Key mirroed sucessfully | 1609021903_1_date2_ran
Key mirroed sucessfully | 1609021903_1_json_ran
Key mirroed sucessfully | 1609021903_1_image_ran
Installation using pypi
pip install redismirror
Installation
$ pip install -r requirements.txt
$ pip install setup.py
Development
This project includes a number of helpers in the Makefile
to streamline common development tasks.
Environment Setup
The following demonstrates setting up and working with a development environment:
### create a virtualenv for development
$ make virtualenv
$ source env/bin/activate
### run redismirror cli application
$ redismirror --help
### run pytest / coverage
$ make test
Releasing to PyPi
Before releasing to PyPi, you must configure your login credentials:
~/.pypirc:
[pypi]
username = YOUR_USERNAME
password = YOUR_PASSWORD
Then use the included helper function via the Makefile
:
$ make dist
$ make dist-upload
Deployments
Docker
Included is a basic Dockerfile
for building and distributing Redis Mirror
,
and can be built with the included make
helper:
$ make docker
$ docker run -it redismirror --help
Extra
To Generate sample data for your test use the below command:
cd tests/Generator/;bash SampleDataInserter.sh
The following commands are also not logged:
- AUTH
- EXEC
- HELLO
- QUIT
Cost of running MONITOR Because MONITOR streams back all commands, its use comes at a cost. The following (totally unscientific) benchmark numbers illustrate what the cost of running MONITOR can be.
Benchmark result without MONITOR running:
$ src/redis-benchmark -c 10 -n 100000 -q
PING_INLINE: 101936.80 requests per second
PING_BULK: 102880.66 requests per second
SET: 95419.85 requests per second
GET: 104275.29 requests per second
INCR: 93283.58 requests per second
Benchmark result with MONITOR running (redis-cli monitor > /dev/null):
$ src/redis-benchmark -c 10 -n 100000 -q
PING_INLINE: 58479.53 requests per second
PING_BULK: 59136.61 requests per second
SET: 41823.50 requests per second
GET: 45330.91 requests per second
INCR: 41771.09 requests per second
In this particular case, running a single MONITOR client can reduce the throughput by more than 50%. Running more MONITOR clients will reduce throughput even more.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for redismirror-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6be4af6237cd194388cfb0fc5fe03d1ac2c269da4879334046a763bce20aefdb |
|
MD5 | 655f3a036660edc78708fe283c35738f |
|
BLAKE2b-256 | 6e2b63e5591601bb6592aafce442fec39ccaefb7ea02258a9b218e48547cac5b |