jsoncache

Python cache control for cloud storage models

This library exposes a multithreaded JSON object loader that support Amazon S3 and Google Cloud Storage.

Why do I care?

Because loading JSON files from the cloud is more annoying than you realize.

• Sometimes you're gonna get errors - log those errors.
• Sometimes you're going to have compressed JSON blobs because Google Cloud Storage has unmanageable timeouts for uploads (https://github.com/googleapis/python-storage/issues/74)
• You want your application to behave as if read errors from the cloud weren't a problem, but you want those errors to show up in logging.

Quick Start

1. Import the ThreadedObjectCache class.
2. Instantiate it passing in the cloud type, bucket, path and time to live in seconds.
3. Call .get() on the ThreadedObjectCache instace.

You can optionally pass in a custom implementation of the time module to override how time.time() works.

You can optionally pass in a custom callable transformer that will apply the transformer function to the data before it's returned. Typical use cases might involve initializing a sklearn model.

You can optionally pass in block_until_cached=True so that the constructor will block until a model is loaded successfully from the network.

All background threads are marked as daemon threads so using this code won't cause your application to wait for thread death.

Python 3.7.8 | packaged by conda-forge | (default, Jul 31 2020, 02:37:09)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.17.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from jsoncache import *

In [2]: t = ThreadedObjectCache('s3', 'telemetry-parquet', 'taar/similarity/lr_curves.json', 10)

In [3]: 2020-08-05 16:07:14,369 - botocore.credentials - INFO - Found credentials in environment variables.
In [3]:

In [3]: t.get()
Out[3]:
[[0.0, [0.029045735469752962, 0.02468400347868071]],
 [0.005000778819764661, [0.029530930135620918, 0.025088940785616222]],
 ...