This is a pre-production deployment of Warehouse, however changes made here WILL affect the production instance of PyPI.
Latest Version Dependencies status unknown Test status unknown Test coverage unknown
Project Description

Lock context manager implemented via redis SETNX/BLPOP.

  • Free software: BSD license

Interface targeted to be exactly like threading.Lock.

Usage

Because we don’t want to require users to share the lock instance across processes you will have to give them names. Eg:

conn = StrictRedis()
with redis_lock.Lock(conn, "name-of-the-lock"):
    print("Got the lock. Doing some work ...")
    time.sleep(5)

Eg:

lock = redis_lock.Lock(conn, "name-of-the-lock")
if lock.acquire(blocking=False):
    print("Got the lock.")
else:
    print("Someone else has the lock.")

You can also associate an identifier along with the lock so that it can be retrieved later by the same process, or by a different one. This is useful in cases where the application needs to identify the lock owner (find out who currently owns the lock). Eg:

import socket
host_id = "owned-by-%s" % socket.gethostname()
lock = redis_lock.Lock(conn, "name-of-the-lock", id=host_id)
if lock.acquire(blocking=False):
    print("Got the lock.")
else:
    if lock.get_owner_id() == host_id:
        print("I already acquired this in another process.")
    else:
        print("The lock is held on another machine.")

Avoid dogpile effect in django

The dogpile is also known as the thundering herd effect or cache stampede. Here’s a pattern to avoid the problem without serving stale data. The work will be performed a single time and every client will wait for the fresh data.

To use this you will need django-redis, however, python-redis-lock provides you a cache backend that has a cache method for your convenience. Just install python-redis-lock like this:

pip install "python-redis-lock[django]"

Now put something like this in your settings:

CACHES = {
    'default': {
        'BACKEND': 'redis_lock.django_cache.RedisCache',
        'LOCATION': 'redis://127.0.0.1:6379/1',
        'OPTIONS': {
            'CLIENT_CLASS': 'django_redis.client.DefaultClient'
        }
    }
}

Note

If using a django-redis < 3.8.x, you’ll probably need redis_cache which has been deprecated in favor to django_redis. The redis_cache module is removed in django-redis versions > 3.9.x. See django-redis notes.

This backend just adds a convenient .lock(name, expire=None) function to django-redis’s cache backend.

You would write your functions like this:

from django.core.cache import cache

def function():
    val = cache.get(key)
    if val:
        return val
    else:
        with cache.lock(key):
            val = cache.get(key)
            if val:
                return val
            else:
                # DO EXPENSIVE WORK
                val = ...

                cache.set(key, value)
                return val

Troubleshooting

In some cases, the lock remains in redis forever (like a server blackout / redis or application crash / an unhandled exception). In such cases, the lock is not removed by restarting the application. One solution is to turn on the auto_renewal parameter in combination with expire to set a time-out on the lock, but let Lock() automatically keep resetting the expire time while your application code is executing:

# Get a lock with a 60-second lifetime but keep renewing it automatically
# to ensure the lock is held for as long as the Python process is running.
with redis_lock.Lock('my-lock', expire=60, auto_renewal=True):
    # Do work....

Another solution is to use the reset_all() function when the application starts:

# On application start/restart
import redis_lock
redis_lock.reset_all()

Alternativelly, you can reset individual locks via the reset method.

Use these carefully, if you understand what you do.

Features

  • based on the standard SETNX recipe
  • optional expiry
  • optional timeout
  • optional lock renewal (use a low expire but keep the lock active)
  • no spinloops at acquire

Implementation

redis_lock will use 2 keys for each lock named <name>:

  • lock:<name> - a string value for the actual lock
  • lock-signal:<name> - a list value for signaling the waiters when the lock is released

This is how it works:

Development

To run the all tests run:

tox

Requirements

OS:Any
Runtime:Python 2.7, 3.3 or later, or PyPy
Services:Redis 2.6.12 or later.

Similar projects

Changelog

3.2.0 (2016-10-29)

  • Changed the signal key cleanup operation do be done without any expires. This prevents lingering keys around for some time. Contributed by Andrew Pashkin in #38.
  • Allow locks with given id to acquire. Previously it assumed that if you specify the id then the lock was already acquired. See #44 and #39.
  • Allow using other redis clients with a strict=False. Normally you’re expected to pass in an instance of redis.StrictRedis.
  • Added convenience method locked_get_or_set to Django cache backend.

3.1.0 (2016-04-16)

  • Changed the auto renewal to automatically stop the renewal thread if lock gets garbage collected. Contributed by Andrew Pashkin in #33.

3.0.0 (2016-01-16)

  • Changed release so that it expires signal-keys immediately. Contributed by Andrew Pashkin in #28.
  • Resetting locks (reset or reset_all) will release the lock. If there’s someone waiting on the reset lock now it will acquire it. Contributed by Andrew Pashkin in #29.
  • Added the extend method on Lock objects. Contributed by Andrew Pashkin in #24.
  • Documentation improvements on release method. Contributed by Andrew Pashkin in #22.
  • Fixed acquire(block=True) handling when expire option was used (it wasn’t blocking indefinitely). Contributed by Tero Vuotila in #35.
  • Changed release to check if lock was acquired with he same id. If not, NotAcquired will be raised. Previously there was just a check if it was acquired with the same instance (self._held). BACKWARDS INCOMPATIBLE
  • Removed the force option from release - it wasn’t really necessary and it only encourages sloppy programming. See #25. BACKWARDS INCOMPATIBLE
  • Dropped tests for Python 2.6. It may work but it is unsupported.

2.3.0 (2015-09-27)

  • Added the timeout option. Contributed by Victor Torres in #20.

2.2.0 (2015-08-19)

  • Added the auto_renewal option. Contributed by Nick Groenen in #18.

2.1.0 (2015-03-12)

  • New specific exception classes: AlreadyAcquired and NotAcquired.
  • Slightly improved efficiency when non-waiting acquires are used.

2.0.0 (2014-12-29)

  • Rename Lock.token to Lock.id. Now only allowed to be set via constructor. Contributed by Jardel Weyrich in #11.

1.0.0 (2014-12-23)

  • Fix Django integration. (reported by Jardel Weyrich)
  • Reorganize tests to use py.test.
  • Add test for Django integration.
  • Add reset_all functionality. Contributed by Yokotoka in #7.
  • Add Lock.reset functionality.
  • Expose the Lock.token attribute.

0.1.2 (2013-11-05)

  • ?

0.1.1 (2013-10-26)

  • ?

0.1.0 (2013-10-26)

  • ?

0.0.1 (2013-10-25)

  • First release on PyPI.
Release History

Release History

3.2.0

This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

3.1.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

3.0.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

2.3.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

2.2.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

2.1.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

2.0.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

1.0.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.1.2

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.1.1

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.1.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.0.1

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

TODO: Brief introduction on what you do with files - including link to relevant help section.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
python_redis_lock-3.2.0-py2.py3-none-any.whl (14.2 kB) Copy SHA256 Checksum SHA256 py2.py3 Wheel Oct 29, 2016
python-redis-lock-3.2.0.tar.gz (159.6 kB) Copy SHA256 Checksum SHA256 Source Oct 29, 2016

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS HPE HPE Development Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting