Redis Shard
A Redis sharding implementation.
Redis is great. It’s fast, lightweight and easy to use. But when we want to store a mass of data into one single instance, we may encounter some problems such as performance degradation and slow recovery and we need to scale it.
Usage
First, Create an RedisShardAPI instance with multiple nodes, node name must be unique:
from redis_shard.shard import RedisShardAPI
servers = [
{'name': 'server1', 'host': '127.0.0.1', 'port': 10000, 'db': 0},
{'name': 'server2', 'host': '127.0.0.1', 'port': 11000, 'db': 0},
{'name': 'server3', 'host': '127.0.0.1', 'port': 12000, 'db': 0},
]
client = RedisShardAPI(servers, hash_method='md5')
Then, you can access the Redis cluster as you use redis-py:
client.set('test', 1)
client.get('test') # 1
client.zadd('testset', 'first', 1)
client.zadd('testset', 'second', 2)
client.zrange('testset', 0, -1) # [first, second]
Config Details
There are three parameters servers, hash_method and sentinel in the class redis_shard.shard.RedisShardAPI.
servers is a list. Each element in it should be a dict or a URL schema.
dict:
[ {'name': 'server1', 'host': '127.0.0.1', 'port': 10000, 'db': 0}, {'name': 'server2', 'host': '127.0.0.1', 'port': 11000, 'db': 0, 'max_connections': 50}, {'name': 'server3', 'host': '127.0.0.1', 'port': 12000, 'db': 0}, ]URL schema:
[ 'redis://127.0.0.1:10000/0?name=node1', 'redis://127.0.0.1:11000/0?name=node2&max_connections=50', 'redis://127.0.0.1:12000/0?name=node3' ]
If the following parameter sentinel is enabled, only name is needed for the servers config.
hash_method is a string which indicate the method of generating the hash key of the consistent hash ring. The default value is crc32. It also supports md5 and sha1.
sentinel is the config for Redis Sentinel. With the sentinel support, redis-shard will do read/write splitting. Config is like this:
{"hosts": [('localhost', 26379)], "socket_timeout": 0.1}
Limitations
Redis Shard dose not support all Redis commands.
As mentioned above, Redis Shard does not support all multi-keys commands crossing different nodes, you have to use Hash Tag to work with those commands.
Redis Shard does not have any replication mechanism.
How it Works
Redis Shard is basically inspired by this article.
Change History
0.3.7 (2021-02-14)
Fix hash tags
0.3.6 (2020-10-15)
Made Redis URL parsing smarter (@huntrax11 )
Compact with python 3.8
Compact with redis-py 3.0.0+
0.3.5 (2017-10-25)
Allow for bytes based keys in Python3 (@gmr)
Fix crc-32 calculation per Python
0.3.4 (2016-07-19)
fix md5 hash_method occus NameError on python3
0.3.3 (2016-03-18)
max_connections can now be set when instantiating client instance, and has a default value 100.
0.3.2 (2015-06-02)
add strict redis support
0.3.1 (2015-02-02)
transaction support
0.3.0 (2014-11-20)
shard key generate function now support md5 and sha1 .
add Redis Sentinel support
0.2.4 (2014-06-26)
remove gevent dependency
0.2.3 (2014-06-02)
better pipeline support
0.2.1 (2013-08-07)
add evel method
0.2.0 (2013-06-02)
add python3 support
0.1.11 (2013-05-06)
add mset support
0.1.10 (2013-04-13)
add mget support ,thks to @Yuekui
0.1.9 (2013-03-04)
add an reshard example
tidy the pipeline code
add more shard methods
0.1.8 (2013-01-21)
add append and getrange method, thks @simon-liu
0.1.7 (2012-11-19)
use new redis url config instead of dict
0.1.5 (2012-07-16)
Add many new methods, support socket_timeout and password
0.1.4 (2011-07-20)
modify hash key algor, support suffix match, thks to dkong
support more redis methods, include keys.
0.1.3 (2011-06-20)
support 2.4.X version of redis-py
0.1.2 (2011-06-01)
add MANIFEST.in file
0.1.1 (2011-05-29)
create hashring use server’s name config.
0.1 (2011-05-28)
first version
Metadata
Release files for redis-shard 0.3.7
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| redis-shard-0.3.7.tar.gz | 13.8 kB | Without provenance | Details |
Release files / redis-shard-0.3.7.tar.gz
| Download URL | redis-shard-0.3.7.tar.gz |
|---|---|
| Size | 13.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4d3d9dffce7f36c664232a76c7685da9ae9e5f5cc367034bc3c5ad873e920df5
|
|
BLAKE2b-256 checksum How to use checksums |
58cacb2c1529225e3deee57d769cd15c33e65ba99d6cc2a334f9af26cec293a7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.50.2 CPython/3.8.1
|
Security for redis-shard 0.3.7
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.