Redis Sharding API
Project description
Redis Shard
###########
.. image:: https://img.shields.io/travis/zhihu/redis-shard.svg?style=flat
:target: https://travis-ci.org/zhihu/redis-shard
:alt: Build Status
.. image:: https://pypip.in/version/redis-shard/badge.svg?style=flat
:target: https://pypi.python.org/pypi/redis-shard
:alt: Latest Version
.. image:: https://pypip.in/py_versions/redis-shard/badge.svg?style=flat
:target: https://pypi.python.org/pypi/redis-shard
:alt: Supported Python versions
.. image:: https://pypip.in/license/redis-shard/badge.svg?style=flat
:target: https://pypi.python.org/pypi/redis-shard
:alt: License
A Redis sharding implementation.
Redis is great. It's fast, lightweight and easy to use. But when we want to store
a mass of data into one single instance, we may encounter some problems such as performance
degradation and slow recovery and we need to scale it.
Usage
=====
First, Create an RedisShardAPI instance with multiple nodes, node ``name`` **must be unique**::
from redis_shard.shard import RedisShardAPI
servers = [
{'name': 'server1', 'host': '127.0.0.1', 'port': 10000, 'db': 0},
{'name': 'server2', 'host': '127.0.0.1', 'port': 11000, 'db': 0},
{'name': 'server3', 'host': '127.0.0.1', 'port': 12000, 'db': 0},
]
client = RedisShardAPI(servers, hash_method='md5')
Then, you can access the Redis cluster as you use `redis-py <https://github.com/andymccurdy/redis-py>`_::
client.set('test', 1)
client.get('test') # 1
client.zadd('testset', 'first', 1)
client.zadd('testset', 'second', 2)
client.zrange('testset', 0, -1) # [first, second]
Hash Tags
---------
If you want to store specific keys on one node for some reason (such as you prefer single instance pipeline, or
you want to use multi-keys command such as ``sinter``), you should use Hash Tags::
client.set('foo', 2)
client.set('a{foo}', 5)
client.set('b{foo}', 5)
client.set('{foo}d', 5)
client.set('d{foo}e', 5)
client.get_server_name('foo') == client.get_server_name('a{foo}') == client.get_server_name('{foo}d') \
== client.get_server_name('d{foo}e') # True
The string in a braces of a key is the Hash Tag of the key. The hash of a Hash Tag will be treated the hash of the key.
So, keys ``foo``, ``bar{foo}`` and ``b{foo}ar`` will be sotred in the same node.
.. note:: Hash Tags are not supported with ``bytes`` keys in Python 3.
Tag method
~~~~~~~~~~~
Just add ``tag_`` prefix, you can use any of the normal redis method on the same hash tag::
client.tag_mget("{user:1}question1", "{user:1}question2")
client.tag_delete("{user:1}question1", "{user:1}question2")
Multi-keys method
~~~~~~~~~~~~~~~~~~
Only support ``mget``, ``mset`` and ``flushdb``.
Config Details
--------------
There are three parameters ``servers``, ``hash_method`` and ``sentinel`` in the :class:`redis_shard.shard.RedisShardAPI`.
``servers`` is a list. Each element in it should be a dict or a URL schema.
- dict::
[
{'name': 'server1', 'host': '127.0.0.1', 'port': 10000, 'db': 0},
{'name': 'server2', 'host': '127.0.0.1', 'port': 11000, 'db': 0, 'max_connections': 50},
{'name': 'server3', 'host': '127.0.0.1', 'port': 12000, 'db': 0},
]
- URL schema::
[
'redis://127.0.0.1:10000/0?name=node1',
'redis://127.0.0.1:11000/0?name=node2&max_connections=50',
'redis://127.0.0.1:12000/0?name=node3'
]
If the following parameter ``sentinel`` is enabled, only **name** is needed for the ``servers`` config.
``hash_method`` is a string which indicate the method of generating the hash key of the consistent hash ring.
The default value is **crc32**. It also supports **md5** and **sha1**.
``sentinel`` is the config for `Redis Sentinel <http://redis.io/topics/sentinel>`_. With the sentinel support, redis-shard
will do read/write splitting. Config is like this::
{"hosts": [('localhost', 26379)], "socket_timeout": 0.1}
Limitations
===========
* Redis Shard dose not support all Redis commands.
* As mentioned above, Redis Shard does not support all multi-keys commands crossing different nodes,
you have to use Hash Tag to work with those commands.
* Redis Shard does not have any replication mechanism.
How it Works
============
Redis Shard is basically inspired by `this article <http://oldblog.antirez.com/post/redis-presharding.html>`_.
Change History
==============
0.3.5 (2017-10-25)
------------------
- Allow for bytes based keys in Python3 (@gmr)
- Fix crc-32 calculation per Python
0.3.4 (2016-07-19)
------------------
- fix md5 hash_method occus NameError on python3
0.3.3 (2016-03-18)
-------------------
- max_connections can now be set when instantiating client instance, and has a default value 100.
0.3.2 (2015-06-02)
--------------------
- add strict redis support
0.3.1 (2015-02-02)
------------------
- transaction support
0.3.0 (2014-11-20)
------------------
- shard key generate function now support **md5** and **sha1** .
- add Redis Sentinel support
0.2.4 (2014-06-26)
------------------
- remove gevent dependency
0.2.3 (2014-06-02)
------------------
- better pipeline support
0.2.1 (2013-08-07)
------------------
- add evel method
0.2.0 (2013-06-02)
------------------
- add python3 support
0.1.11 (2013-05-06)
-------------------
- add mset support
0.1.10 (2013-04-13)
-------------------
- add mget support ,thks to @Yuekui
0.1.9 (2013-03-04)
------------------
- add an reshard example
- tidy the pipeline code
- add more shard methods
0.1.8 (2013-01-21)
------------------
- add `append` and `getrange` method, thks @simon-liu
0.1.7 (2012-11-19)
------------------
- use new redis url config instead of dict
0.1.5 (2012-07-16)
------------------
- Add many new methods, support socket_timeout and password
0.1.4 (2011-07-20)
------------------
- modify hash key algor, support suffix match, thks to dkong
- support more redis methods, include `keys`.
0.1.3 (2011-06-20)
------------------
- support 2.4.X version of redis-py
0.1.2 (2011-06-01)
------------------
- add MANIFEST.in file
0.1.1 (2011-05-29)
------------------
- create hashring use server's name config.
0.1 (2011-05-28)
----------------
- first version
###########
.. image:: https://img.shields.io/travis/zhihu/redis-shard.svg?style=flat
:target: https://travis-ci.org/zhihu/redis-shard
:alt: Build Status
.. image:: https://pypip.in/version/redis-shard/badge.svg?style=flat
:target: https://pypi.python.org/pypi/redis-shard
:alt: Latest Version
.. image:: https://pypip.in/py_versions/redis-shard/badge.svg?style=flat
:target: https://pypi.python.org/pypi/redis-shard
:alt: Supported Python versions
.. image:: https://pypip.in/license/redis-shard/badge.svg?style=flat
:target: https://pypi.python.org/pypi/redis-shard
:alt: License
A Redis sharding implementation.
Redis is great. It's fast, lightweight and easy to use. But when we want to store
a mass of data into one single instance, we may encounter some problems such as performance
degradation and slow recovery and we need to scale it.
Usage
=====
First, Create an RedisShardAPI instance with multiple nodes, node ``name`` **must be unique**::
from redis_shard.shard import RedisShardAPI
servers = [
{'name': 'server1', 'host': '127.0.0.1', 'port': 10000, 'db': 0},
{'name': 'server2', 'host': '127.0.0.1', 'port': 11000, 'db': 0},
{'name': 'server3', 'host': '127.0.0.1', 'port': 12000, 'db': 0},
]
client = RedisShardAPI(servers, hash_method='md5')
Then, you can access the Redis cluster as you use `redis-py <https://github.com/andymccurdy/redis-py>`_::
client.set('test', 1)
client.get('test') # 1
client.zadd('testset', 'first', 1)
client.zadd('testset', 'second', 2)
client.zrange('testset', 0, -1) # [first, second]
Hash Tags
---------
If you want to store specific keys on one node for some reason (such as you prefer single instance pipeline, or
you want to use multi-keys command such as ``sinter``), you should use Hash Tags::
client.set('foo', 2)
client.set('a{foo}', 5)
client.set('b{foo}', 5)
client.set('{foo}d', 5)
client.set('d{foo}e', 5)
client.get_server_name('foo') == client.get_server_name('a{foo}') == client.get_server_name('{foo}d') \
== client.get_server_name('d{foo}e') # True
The string in a braces of a key is the Hash Tag of the key. The hash of a Hash Tag will be treated the hash of the key.
So, keys ``foo``, ``bar{foo}`` and ``b{foo}ar`` will be sotred in the same node.
.. note:: Hash Tags are not supported with ``bytes`` keys in Python 3.
Tag method
~~~~~~~~~~~
Just add ``tag_`` prefix, you can use any of the normal redis method on the same hash tag::
client.tag_mget("{user:1}question1", "{user:1}question2")
client.tag_delete("{user:1}question1", "{user:1}question2")
Multi-keys method
~~~~~~~~~~~~~~~~~~
Only support ``mget``, ``mset`` and ``flushdb``.
Config Details
--------------
There are three parameters ``servers``, ``hash_method`` and ``sentinel`` in the :class:`redis_shard.shard.RedisShardAPI`.
``servers`` is a list. Each element in it should be a dict or a URL schema.
- dict::
[
{'name': 'server1', 'host': '127.0.0.1', 'port': 10000, 'db': 0},
{'name': 'server2', 'host': '127.0.0.1', 'port': 11000, 'db': 0, 'max_connections': 50},
{'name': 'server3', 'host': '127.0.0.1', 'port': 12000, 'db': 0},
]
- URL schema::
[
'redis://127.0.0.1:10000/0?name=node1',
'redis://127.0.0.1:11000/0?name=node2&max_connections=50',
'redis://127.0.0.1:12000/0?name=node3'
]
If the following parameter ``sentinel`` is enabled, only **name** is needed for the ``servers`` config.
``hash_method`` is a string which indicate the method of generating the hash key of the consistent hash ring.
The default value is **crc32**. It also supports **md5** and **sha1**.
``sentinel`` is the config for `Redis Sentinel <http://redis.io/topics/sentinel>`_. With the sentinel support, redis-shard
will do read/write splitting. Config is like this::
{"hosts": [('localhost', 26379)], "socket_timeout": 0.1}
Limitations
===========
* Redis Shard dose not support all Redis commands.
* As mentioned above, Redis Shard does not support all multi-keys commands crossing different nodes,
you have to use Hash Tag to work with those commands.
* Redis Shard does not have any replication mechanism.
How it Works
============
Redis Shard is basically inspired by `this article <http://oldblog.antirez.com/post/redis-presharding.html>`_.
Change History
==============
0.3.5 (2017-10-25)
------------------
- Allow for bytes based keys in Python3 (@gmr)
- Fix crc-32 calculation per Python
0.3.4 (2016-07-19)
------------------
- fix md5 hash_method occus NameError on python3
0.3.3 (2016-03-18)
-------------------
- max_connections can now be set when instantiating client instance, and has a default value 100.
0.3.2 (2015-06-02)
--------------------
- add strict redis support
0.3.1 (2015-02-02)
------------------
- transaction support
0.3.0 (2014-11-20)
------------------
- shard key generate function now support **md5** and **sha1** .
- add Redis Sentinel support
0.2.4 (2014-06-26)
------------------
- remove gevent dependency
0.2.3 (2014-06-02)
------------------
- better pipeline support
0.2.1 (2013-08-07)
------------------
- add evel method
0.2.0 (2013-06-02)
------------------
- add python3 support
0.1.11 (2013-05-06)
-------------------
- add mset support
0.1.10 (2013-04-13)
-------------------
- add mget support ,thks to @Yuekui
0.1.9 (2013-03-04)
------------------
- add an reshard example
- tidy the pipeline code
- add more shard methods
0.1.8 (2013-01-21)
------------------
- add `append` and `getrange` method, thks @simon-liu
0.1.7 (2012-11-19)
------------------
- use new redis url config instead of dict
0.1.5 (2012-07-16)
------------------
- Add many new methods, support socket_timeout and password
0.1.4 (2011-07-20)
------------------
- modify hash key algor, support suffix match, thks to dkong
- support more redis methods, include `keys`.
0.1.3 (2011-06-20)
------------------
- support 2.4.X version of redis-py
0.1.2 (2011-06-01)
------------------
- add MANIFEST.in file
0.1.1 (2011-05-29)
------------------
- create hashring use server's name config.
0.1 (2011-05-28)
----------------
- first version
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
redis-shard-0.3.5.tar.gz
(13.0 kB
view hashes)