Some of Hubspot's python cache utils, namely generational caching
Project description
[](https://travis-ci.org/HubSpot/hscacheutils)
A python implementation of [generational caching](http://www.regexprn.com/2011/06/web-application-caching-strategies_05.html).
Three main interfaces are provided:
- Direct methods `gen_cache.get`, `gen_cache.set`, `gen_cache.invalidate`, and `gen_cache.delete`
- A `CustomUseGenCache` instance
- A function decorator `@gen_cache.wrap`
## Direct methods:
```python
from hscacheutils.generational_cache import gen_cache
html = gen_cache.get(('nav', 'nav_portal:user_id'), user_id=1)
gen_cache.set('<html>', ('nav', 'nav_portal:user_id'), user_id=1)
gen_cache.invalidate(('nav', 'nav_portal:user_id'), user_id=1)
```
## A `CustomUseGenCache` instance
The same as the direct methods, but creates an object so that you con't have to keep on passing in the generation names every single time.
```python
custom_cache = CustomUseGenCache([
'customgenz:user_id',
'customgenz:blog_id'
])
blog_id = 17
user_id = 123
key = random.randint(1, 20000000)
val = custom_cache.get(blog_id=blog_id, user_id=123, cache_key=key)
custom_cache.set(value=first_val, blog_id=blog_id, user_id=123, cache_key=key)
custom_cache.invalidate(user_id=123)
custom_cache.delete(blog_id=blog_id, user_id=123, cache_key=key)
```
## The `@gen_cache.wrap` decorator
It can be applied to function, method or classmethod. It is mostly similar to gen_cache.get, but with some additional magic to make your life easier.
### Magic #1: (true for all 3 interfaces)
The contents of "value-based" generations are automatically pulled from the arguments in wrapped function. Eg.
```python
@gen_cache.wrap('project_name', 'foo_per_user_id:user_id')
def foobar(user_id):
...
foobar(53) # Uses 'project_name' and 'foo_per_user_id:53' as generations
foobar(999) # Uses 'project_name' and 'foo_per_user_id:999' as generations
# So when invalidating like so...
gen_cache.invalidate("for_per_user_id:user_id", user_id=999)
foobar(53) # This is still cached
foobar(999) # This has been invalidated
```
So the when foobar is called the ':user_id' part of the value-based generation looks for any
argument named "user_id", then takes its value to create a generation such as "for_per_user_id:53".
This means that the "for_per_user_id" generation is only invalidated on a per-portal basis
### Magic #2: (true for all 3 interfaces)
All of the arguments (not used in value-based generations described above) are automatically appended to the cache key. Eg.
```python
@gen_cache.wrap('whatever')
def foobar(something, another=False):
...
# XXX represents the current counter value of the 'whatever' generation
foobar(1) # Uses a cache key roughly like: "whatever:XXX [1]{another=False}"
foobar(2) # Uses a cache key roughly like: "whatever:XXX [2]{another=False}"
foobar(2, another=True) # Uses a cache key roughly like: "whatever:XXX [2]{another=True}"
# So when invalidating like so...
gen_cache.invalidate("whatever")
foobar(1) # This has been invalidated
foobar(2) # This has been invalidated
foobar(2, another=True) # This has been invalidated
```
If you don't what this behavior for one or more arguments, make sure to put the name of that
argument(s) in the "exclude" option (see below).
### Magic #3: (only true for the decorator)
The cache key will automatically include the current module name, function name, and
line number. So when this function moves to a different file, is renamed, or moves up or down a
few lines, the cache will automatically be invalidated.
(Note, I'm not sure this file/function name magic is worth keeping)
### EXTRAS
The wrapped callable gets `invalidate` methods. Call `invalidate` with
same arguments as function and the result for these arguments will be
invalidated.
### KEYWORD OPTIONS
timeout=3600 (defaults to None) is the number of seconds before this cache should expire
exclude=[...] (defaults to empty list) is all the arguments you do not want to automaticaaly be
a included in the cache key.
log_misses=True (False by default) will print out some debugging into on every cache miss
ignore_locally=True (False by default) will disable this caching when ENV == 'local'
### REAL `gen_cache.wraps` cache key example
[cached]hsdjango.test.test_generational_cache.func_with_lots_of_args:369(['one','two']{'project':1336056824437339,'foobar':'NOThello','user_id':42})
^ ^ ^ ^ ^ ^ ^
| | | | | | |
prefix module name func name line # | generation & current counter value |
| |
non-excluded positional args non-excluded keyword args
Gotcha #1: Be careful to use either "self" or "cls" as the first argument name when wrapping
methods and classmethods. This code relies on those names (see _func_type) to automatically
chop off the first argument from the cache key.
_Note: based on (and built re-using) django-cache-utils._
A python implementation of [generational caching](http://www.regexprn.com/2011/06/web-application-caching-strategies_05.html).
Three main interfaces are provided:
- Direct methods `gen_cache.get`, `gen_cache.set`, `gen_cache.invalidate`, and `gen_cache.delete`
- A `CustomUseGenCache` instance
- A function decorator `@gen_cache.wrap`
## Direct methods:
```python
from hscacheutils.generational_cache import gen_cache
html = gen_cache.get(('nav', 'nav_portal:user_id'), user_id=1)
gen_cache.set('<html>', ('nav', 'nav_portal:user_id'), user_id=1)
gen_cache.invalidate(('nav', 'nav_portal:user_id'), user_id=1)
```
## A `CustomUseGenCache` instance
The same as the direct methods, but creates an object so that you con't have to keep on passing in the generation names every single time.
```python
custom_cache = CustomUseGenCache([
'customgenz:user_id',
'customgenz:blog_id'
])
blog_id = 17
user_id = 123
key = random.randint(1, 20000000)
val = custom_cache.get(blog_id=blog_id, user_id=123, cache_key=key)
custom_cache.set(value=first_val, blog_id=blog_id, user_id=123, cache_key=key)
custom_cache.invalidate(user_id=123)
custom_cache.delete(blog_id=blog_id, user_id=123, cache_key=key)
```
## The `@gen_cache.wrap` decorator
It can be applied to function, method or classmethod. It is mostly similar to gen_cache.get, but with some additional magic to make your life easier.
### Magic #1: (true for all 3 interfaces)
The contents of "value-based" generations are automatically pulled from the arguments in wrapped function. Eg.
```python
@gen_cache.wrap('project_name', 'foo_per_user_id:user_id')
def foobar(user_id):
...
foobar(53) # Uses 'project_name' and 'foo_per_user_id:53' as generations
foobar(999) # Uses 'project_name' and 'foo_per_user_id:999' as generations
# So when invalidating like so...
gen_cache.invalidate("for_per_user_id:user_id", user_id=999)
foobar(53) # This is still cached
foobar(999) # This has been invalidated
```
So the when foobar is called the ':user_id' part of the value-based generation looks for any
argument named "user_id", then takes its value to create a generation such as "for_per_user_id:53".
This means that the "for_per_user_id" generation is only invalidated on a per-portal basis
### Magic #2: (true for all 3 interfaces)
All of the arguments (not used in value-based generations described above) are automatically appended to the cache key. Eg.
```python
@gen_cache.wrap('whatever')
def foobar(something, another=False):
...
# XXX represents the current counter value of the 'whatever' generation
foobar(1) # Uses a cache key roughly like: "whatever:XXX [1]{another=False}"
foobar(2) # Uses a cache key roughly like: "whatever:XXX [2]{another=False}"
foobar(2, another=True) # Uses a cache key roughly like: "whatever:XXX [2]{another=True}"
# So when invalidating like so...
gen_cache.invalidate("whatever")
foobar(1) # This has been invalidated
foobar(2) # This has been invalidated
foobar(2, another=True) # This has been invalidated
```
If you don't what this behavior for one or more arguments, make sure to put the name of that
argument(s) in the "exclude" option (see below).
### Magic #3: (only true for the decorator)
The cache key will automatically include the current module name, function name, and
line number. So when this function moves to a different file, is renamed, or moves up or down a
few lines, the cache will automatically be invalidated.
(Note, I'm not sure this file/function name magic is worth keeping)
### EXTRAS
The wrapped callable gets `invalidate` methods. Call `invalidate` with
same arguments as function and the result for these arguments will be
invalidated.
### KEYWORD OPTIONS
timeout=3600 (defaults to None) is the number of seconds before this cache should expire
exclude=[...] (defaults to empty list) is all the arguments you do not want to automaticaaly be
a included in the cache key.
log_misses=True (False by default) will print out some debugging into on every cache miss
ignore_locally=True (False by default) will disable this caching when ENV == 'local'
### REAL `gen_cache.wraps` cache key example
[cached]hsdjango.test.test_generational_cache.func_with_lots_of_args:369(['one','two']{'project':1336056824437339,'foobar':'NOThello','user_id':42})
^ ^ ^ ^ ^ ^ ^
| | | | | | |
prefix module name func name line # | generation & current counter value |
| |
non-excluded positional args non-excluded keyword args
Gotcha #1: Be careful to use either "self" or "cls" as the first argument name when wrapping
methods and classmethods. This code relies on those names (see _func_type) to automatically
chop off the first argument from the cache key.
_Note: based on (and built re-using) django-cache-utils._
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
hscacheutils-0.1.8.tar.gz
(10.1 kB
view details)
File details
Details for the file hscacheutils-0.1.8.tar.gz.
File metadata
- Download URL: hscacheutils-0.1.8.tar.gz
- Upload date:
- Size: 10.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 4ad2e24b5672e05255f217867e627f9f010b3fe36556ea1a7c524f2d070364dc |
|
| MD5 | 6d2a5d0e4a5816184c03b7ba6931d00d |
|
| BLAKE2b-256 | 4cd43ba0166909a753dcbcce8277e1e298a9953c1722dbbab39d713f0d96bb71 |
