A python class for using redis, or other key value stores, as a dictionary and caching the values locally for read heavy workloads. Heavily inspired by pylru.
Usage
The idea is that Deleting or Updating keys in an instance of PubSubRedisDict or PubSubCacheManager will update the matching cached keys in all instances of PubSubCacheManager. PubSubCacheManager will therefor maintain a cache of recently used keys using an lru or just a straight up dict(). This will reduce the round trip latency and network overhead for any reads of the cached keys.
RedisDict and RedisDict should work with instances of redis.StrictRedis or rediscluster.StrictRedisCluster. Use the prefix for managing redis key namespaces.
RedisDict
Just like a normal dictionary, but networked. Initialisation wont take a dictionary or iterable for now as it need connection and namespace information.
rc = StrictRedisCluster(startup_nodes=[{"host": "redis", "port": "6379"}])
reddict = RedisDict(rc, 'namespace')
# you can set
reddict[1] = 1
reddict[2] = [1,2,3]
reddict['hello'] = 'world'
reddict[('complex',1)] = {'I': {'Am': {'Quite': ['a', 'complex', {'object': {} }]}}}
# get somewhere else
reddict[1]
reddict['1'] # note its the same as reddict[1]
reddict[('complex',1)]
reddict["('complex', 1)"] # the key is str(('complex',1))
# delete
del reddict[1]
# .. ect
PubSubRedisDict
Like RedisDict but will publish key update and delete events to a <namespace>/[update|delete] channel.
redpubsub = PubSubRedisDict(rc, 'namespace')
# ect as before
PubSubCacheManager
Like pylry.WriteThroughCacheManager but updates cache keys from store when it receives a message from the <namespace>/[update|delete] channel.
cache = pylru.lrucache(10) # maybe more than 10
redstore = PubSubRedisDict(rc, 'namespace')
redcache = PubSubCacheManager(redstore, cache)
# ect as before
# see the cache
print dict(redcache.cache)
Further uses
You can hook up RedisDict or PubSubRedisDict to pylru.WriteBackCacheManager to get a Redis backed dictionary which only writes to Redis on ‘flush’ or when the item pops off the lru for write intensive workloads. However a lot more work would need to be done to add the pubsub mechanism as there difficult cases to consider, such as what happens when the cache is dirty and we get notified that the store key is updated?
Limitations
all keys are strings.
msgpack is used to marshal objects to redis, so msgpack object limitations apply. Though you can monkey patch the modules loads and dumps method if you like.
publish will publish to all consuming dictionary instances, there is no partitioning, so writes and updates are expensive. You could come up with a partitioning strategy to improve this.
The published items eventually end up in the watched cash. There may be a time lag between a client publishing a change and the key updating in another clients cache.
References
Metadata
Release files for redis-pubsub-dict 0.0.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| redis_pubsub_dict-0.0.2.tar.gz | 4.9 kB | Without provenance | Details for redis_pubsub_dict-0.0.2.tar.gz |
Release files / redis_pubsub_dict-0.0.2.tar.gz
| Download URL | redis_pubsub_dict-0.0.2.tar.gz |
|---|---|
| Size | 4.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7229c21e8cb12838d66bc923dffdd54e15f009b0b8816f5ab7564b5d93b664c9
|
|
BLAKE2b-256 checksum How to use checksums |
f927369fd6f17c42b30443b32f57365ec4bc6463d00cf8abdb52aae49690eb9d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Security for redis-pubsub-dict 0.0.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.