Skip to main content

caching

Library for easily caching computations that take a long time to run, e.g., training Keras models.

Installation

  1. Optionally, create and/or activate a virtual environment to install this package in. E.g.,
conda create -n example_env python=3
conda activate example_env
  1. Clone the caching repo to a location of your choice:
git clone https://github.com/klasleino/caching.git
  1. Change into the repo directory and install using pip:
cd caching
pip install -e .

Usage

Caching utility is provided via decorators. The primary decorator is the Cachable decorator found in the cachable package. This decorator is used on a function (either member or non-member) to specify that the value returned by the function should be cached. If the function is called again with the same parameters, the result will be loaded from the cache rather than running the function again.

For example, consider the following code:

from cachable import Cachable

@Cachable(directory='cache')
def f(a, b, c):
    return dict(a=a, b=b, c=c)

# Call 1 (body of `f` is run).
res = f(1, 2, 3)

# Call 2 (loaded from cache).
res = f(1, 2, 3)

# Call 3 (body of `f` is run).
res = f(4, 5, 6)

In the first call, the function, f, will be run, and the result will be saved in a file in a directory called "cache". The file name of the cached result will contain the parameters, a, b, and c and the values passed to them in call 1. In the second call, the function will not be run, because an existing file with the given parameters exists; instead, the result will be loaded from the cached file. Finally, in the third call, the parameters are different, thus the function will be run again, and a new cache entry made.

Cached Objects

Objects returned by Cachable functions are wrapped as a cachable.cached_objects.CachedObject. CachedObjectss can mostly be used the same as their non-wrapped counterparts, but the original object returned by the non-decorated function can be obtained from the obj field of the CachedObject. The CachedObject also keeps track of the parameters used to create the object.

Default Parameters

Parameters taking their default values will not be included in the name of the cached file. This keeps file names shorter, and allows backwards compatibility with previously cached files, provided the default value is chosen appropriately.

Non-primitive Arguments

Any argument with a canonical string representation can be passed as an argument to a Cachable function. The argument, arg, is taken to be an unseen value if str(arg) does not match the string representation of any previous values passed. This may present a problem for generic objects, as two objects may be the same, while str(obj1) != str(obj2).

Any instance of CachedObject has a canonical string representation, so CachedObjects can be given as parameters to other cached objects.

For other custom objects, the CachableParam decorator in the cachable package can be used. This gives the decorated object a canonical string representation based on the arguments passed to the object's __init__ function. For example, a custom object can be used as a parameter to a Cachable function using the following code:

from cachable import Cachable, CachableParam

@CachableParam()
class P(object):
    def __init__(a, b, c):
        self.a = a
        self.b = b
        self.c = c

@Cachable(directory='cache')
def f(p):
    return dict(a=p.a, b=p.b, c=p.c)

# Call 1 (body of `f` is run).
res = f(P(1, 2, 3))

# Call 2 (loaded from cache).
res = f(P(1, 2, 3))

Note that the behavior of the caching may not be correct if the CachableParam is modified after it is constructed; therefore this should only be used on objects that are treated as immutable.

Hidden Parameters

If a Cachable function takes a parameter that for whatever reason should not affect the cached file name, the parameter can be hidden by beginning its name with an underscore, e.g.,

from cachable import Cachable

@Cachable(directory='cache')
def f(a, b, c, _msg):
    print(_msg)
    return dict(a=a, b=b, c=c)

# Call 1 (body of `f` is run).
res = f(1, 2, 3, 'hello')

# Call 2 (loaded from cache).
res = f(1, 2, 3, 'world')

Refreshing

If for whatever reason, it is desired for the body of a Cachable function to be rerun with a set of parameters previously used, a special parameter called _refresh can be used, e.g.,

from cachable import Cachable

@Cachable(directory='cache')
def f(a, b, c):
    return dict(a=a, b=b, c=c)

# Call 1 (body of `f` is run).
res = f(1, 2, 3)

# Call 2 (body of `f` is run).
res = f(1, 2, 3, _refresh=True)

Loaders

By default, the results of Cachable functions are saved using python's pickle utility. If another format is preferred, an alternative loader can be used. Several default implementations of alternative loaders can be found in the cachable.loaders package. For example, to cache a Keras model, the following could be used to save the resulting model as an h5 file using Keras' model.save():

from cachable import Cachable
from cachable.loaders import KerasModelLoader

@Cachable(name='my_cached_model', directory='cache', loader=KerasModelLoader())
def create_and_train_model(**model_hyperparameters):
    # Create and train a model...
    return model

Metadata

Release files for cachable 0.0.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for cachable 0.0.1
File Size Uploaded Provenance Actions
cachable-0.0.1.tar.gz 9.1 kB Without provenance Details for cachable-0.0.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for cachable 0.0.1
File Interpreter ABI Platform Actions
cachable-0.0.1-py3-none-any.whl Python 3 none any Details for cachable-0.0.1-py3-none-any.whl

Total release size: 18.0 kB

Release files / cachable-0.0.1.tar.gz

Download URL cachable-0.0.1.tar.gz
Size 9.1 kB
Tags Source
SHA-256 checksum
How to use checksums
9112caa2d70db38db8d0c6862a9b75f95a8c072d49739482966009f80d92d7bd
BLAKE2b-256 checksum
How to use checksums
40bee568ee0991b2186f593689c729f6e04b8594d7f55197f5a79f3d4cc63984
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/46.1.3.post20200330 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.8.2

Release files / cachable-0.0.1-py3-none-any.whl

Download URL cachable-0.0.1-py3-none-any.whl
Size 8.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
a1bda4ad372b333b0ec4138a4c24b74543ba1ad11cb60cef6dbca2ccf38522ae
BLAKE2b-256 checksum
How to use checksums
aa9c0659eb88fc7de533a1c434ab3788b2e0a434951bdc2b81328c17210036a4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/46.1.3.post20200330 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.8.2

Security for cachable 0.0.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.0.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page