redpipe 4.0.1

Easy Redis pipelines

RedPipe

Pain-Free redis pipelining in python.

|BuildStatus| |CoverageStatus| |Version| |Python| |Docs|

RedPipe is a wrapper around the pipeline component of redis-py <https://redis-py.readthedocs.io>_. It makes it easy to reduce network round trips when talking to Redis.

For more general information about redis pipelining, see the official redis documentation <https://redis.io/topics/pipelining>_.

Use RedPipe to build pipelined redis calls in a modular reusable way. Rewrite your existing application calls via redis-py into efficient batches with only minimal changes.

Requirements

The redpipe module requires Python 2.7 or higher.

Installation

To install, use pip:

.. code-block::

$ pip install redpipe

or from source:

.. code-block::

$ python setup.py install

Quick Start

To use redpipe, You need to bind your redis client instance to RedPipe. Use the standard redis-py client.

.. code-block:: python

client = redis.Redis()
redpipe.connect_redis(client)

You only need to do this setup once during application bootstrapping.

Using RedPipe is easy. We can pipeline multiple calls to redis and assign the results to variables. This makes pipeline code look and feel similar to the regular interface of redis-py.

.. code-block:: python

with redpipe.pipeline() as pipe:
    foo = pipe.incr('foo')
    bar = pipe.incr('bar)
    pipe.execute()
print([foo, bar])

Documentation

Find RedPipe documentation on Read the Docs <http://redpipe.readthedocs.io/en/latest/>_.

.. |BuildStatus| image:: https://travis-ci.org/72squared/redpipe.svg?branch=master :target: https://app.travis-ci.com/github/72squared/redpipe

.. |CoverageStatus| image:: https://coveralls.io/repos/github/72squared/redpipe/badge.svg?branch=master :target: https://coveralls.io/github/72squared/redpipe?branch=master

.. |Version| image:: https://badge.fury.io/py/redpipe.svg :target: https://badge.fury.io/py/redpipe

.. |Python| image:: https://img.shields.io/badge/python-2.7,3.4,pypy-blue.svg :target: https://pypi.python.org/pypi/redpipe/

.. |Docs| image:: https://readthedocs.org/projects/redpipe/badge/?version=latest :target: http://redpipe.readthedocs.org/en/latest/ :alt: Documentation Status

Release Notes

4.0.1 (December 28, 2022)

Modify Protocol import to allow python 3.7 to work.

4.0.0 (December 27, 2022)

Add support for type hints. Remove any python 2 references. Remove dependency on "six" library now that only python3 is supported.

3.0.1 (December 26, 2022)

Update the docs to reflect deprecation of redis-py-cluster. Use redis-py instead.

3.0.0 (December 26, 2022)

Maintain compatibility with latest version of redis-py, now that it natively supports redis cluster.

2.3.6 (October 30, 2019)

Bugfix. enable_threads and disable_threads behavior were swapped. Added a test to confirm correct behavior.

2.3.5 (July 3, 2019)

Fixing an issue where iterators getting passed in to multi key operations were being treated as a single key.

2.3.4 (June 24, 2019)

StringListField should return an empty string for an empty list encoding. Sending None to the redis results in bad stuff.

2.3.3 (June 21, 2019)

Relaxing validation for fields a little bit and try to just do the right thing. Only blow up if we really can't coerce to what we think is a correct value.

2.3.2 (June 19, 2019)

Make IntegerField a little more forgiving and allow float values to be passed in. Auto-convert to integers. Makes lots of things easier.

2.3.1 (Nov 29, 2018)

Introduce Keyspace.super_pipe property. Allows multiple pipelined functions to share the same parent pipeline. Don't use this unless you know what you're doing. We will work out the documentation as we explore the need for this further and standardize the api.

2.3.0 (Nov 28, 2018)

Bug fix: pickling future objects with a result that evaluates to false causes unpickling to not call setstate. This causes the future to raise an exception on unpickling when trying to use the future result because no state has been set.

In python documentation it states: "Note For new-style classes, if getstate() returns a false value, the setstate() method will not be called."

reference: https://docs.python.org/3/library/pickle.html

The fix is to return a dictionary with the result in getstate and expect one in setstate so the value never evaluates to false.

2.2.0 (Aug 14, 2018)

z[rev]range[byscore] functions should return list of tuples instead of list of lists. fix provided by Max Greenblatt, thanks! This makes the interface more closely match the behavior in redis-py which acts as a reference spec for redpipe commands. This is a minor change and probably won't break anything on the client, but incrementing the minor version, just in case.

2.1.2 (July 17, 2018)

enable key by name as attribute.

2.1.1 (July 13, 2018)

If you set field_attr_on to true on a Struct, you can access fields as attributes of the struct. Not recommended, but there may be certain cases where it can be desirable. Use with caution.

2.1.0 (July 11, 2018)

Support for Strings stored in a sharded Hash. Adding mget on String keyspace data. adding required fields and defaults on struct.

2.0.2 (May 23, 2018)

rename _memberparse to memberparse on Hash class to be consistent with naming conventions of keyparse and valueparse variables. Also allows access of this variable outside the class which is fine.

2.0.1 (March 15, 2018)

ttl on struct objects. documentation fixes.

2.0.0 (March 8, 2018)

to avoid confusion, make the eval command in keyspaces match the interface defined in redis-py.

1.0.4 (September 28, 2017)

adding support for zcount (somehow missed it before).

1.0.3 (August 10, 2017)

fixing a bug in named connections in structs and keyspaces. fixes issue #2

1.0.2 (June 23, 2017)

enable pfcount when using rediscluster.

1.0.1 (May 29, 2017

adding an nx option to struct to allow changes to be set only if the properties don't already exist.

1.0.0 (May 11, 2017

No substantive changes from 1.0.0rc3. Updating notes and removing beta flags.

1.0.0rc3 (May 10, 2017)

Use threads by default when talking to multiple backends in the same pipeline. You can disable this behavior with redpipe.disable_threads().

1.0.0rc2 (May 9, 2017)

Minor changes.

• make the keyspace object call conform to redis-py
• use twine to publish to pypi
• publish wheels

1.0.0rc1 (May 7, 2017)

This marks the first RC. There are a few breaking changes, mostly easily fixed.

• better handling of Nones returned from hmget in Struct
• testing with toxiproxy to simulate slower networks in benchmarks
• using pytest-benchmark tool for benchmark comparisons
• simplifying connections so we can pass in redis or rediscluster
• fixing some compat issues with redis-py interface

0.5.0 (May 5, 2017)

More breaking changes to Struct. Solidifying the api. Making important simplifications. This will make it easier to explain and document.

• Struct and Keyspace: simplifying some variable names
• Struct: support a no_op flag to prevent read/write from redis
• Struct: no kwargs as properties of struct. a dict models it better
• Struct: specify fields to load when instantiating
• Struct: reuse remove logic in the update function for elements set to None
• Simplifying task wait and promise to use the TaskManager directly
• Future: better isinstance and is comparison checks
• make it easier to build docs
• adding Docker support for testing many versions of python

0.4.0 (May 4, 2017)

• by default, don't use transactions
• autocommit flag renamed to autoexec. Breaking change.
• support pickling Struct
• make repr(Struct) more standard
• cleaner connection and pipeline interfaces
• verify redis cluster support with a single-node redis cluster via redislite

0.3.2 (May 3, 2017)

After experimenting with some things, simplifying Struct back down. Some of the methods in Struct will break. Easier to explain with fewer methods and can still do everything I need to.

• cleaner support for items and iteritems in struct
• support for delete in struct
• fixed a bug with deleting multiple keys in Keyspace objects.
• simplification on json serialization detection
• test flake8 on travis
• test with hiredis

This release also improves the documentation on Struct. I hadn't bothered much up until this point. The interface was still solidifying. Starting to get to a stable place there.

0.3.1 (May 2, 2017)

Breaking changes in this release as well. Can only access data from a struct object like you would a dictionary. This is an important step because it disambiguates commands from data. And it enforces one consistent way to access data. All the methods on the Struct give it a dictionary interface. Easier to explain the mental model this way.

• Improvements to redpipe.Struct.
• Documentation improvements.

0.3.0 (April 30, 2017)

BIG REFACTOR. key no longer part of the constructor of Keyspace objects. Instead, you pass the key name to the method. This keeps the api identical in arguments in redis-py. It also allows me to support multi-key operations. This is a breaking change.

• no need for a compat layer, using six
• standardize key, value, member encoding & decoding by reusing Field interface
• key no longer part of the constructor of Keyspace objects

0.2.5 (April 30, 2017)

• support for binary field
• improving encoding and decoding in Keyspaces
• alias iteritems to items on struct
• make fields use duck-typing to validate instead of using isinstance

0.2.4 (April 28, 2017)

• better interface for async enable/disable.
• add ability to talk to multiple redis servers in parallel via threads

0.2.3 (April 27, 2017)

• renaming datatypes to keyspaces. easier to explain.
• moving documentation from readme into docs/ for readthedocs.
• support for ascii field

0.2.2 (April 26, 2017)

• better support and testing of redis cluster
• support for hyperloglog data type
• adding support for more complex field types
• support sortedset lex commands
• support for scanning

0.2.1 (April 24, 2017)

• bug fix: make sure accessing result before ready results in a consistent exception type.
• bug fix: issue when exiting with statement from python cli

0.2.0 (April 24, 2017)

• make the deferred object imitate the underlying result

0.1.1 (April 23, 2017)

• make it possible to typecast fields in the Hash data type
• better support for utf-8
• make result object traceback cleaner

0.1.0 (April 21, 2017)

• better pipelining and task management
• better support for multi pipeline use case

Old Releases

Releases prior to 1.0.0 are considered beta. The api is not officially supported. We make no guarantees about backward compatibility.

Releases less than 0.1.0 in this project are considered early alpha and don't deserve special mention.