Python Simple-Caching
Project description
Simple Caching
Small project to standardize storing (key, value) data for caching purposes.
Supported caching methods:
- Disk
- NpyFS - Numpy array export
- Memory
- DictMemory - keys are stored as dict keys and recalled from memory
Example
from simple_caching.storage import MemoryDict
import numpy as np
import time
def working_fn(image: np.ndarray) -> np.ndarray:
"""Working function that takes on average 0.2s"""
if np.random.rand() <= 0.2:
time.sleep(1)
return (image - image.min()) / (image.max() - image.min())
data = np.random.randn(32, 240, 420, 3).astype(np.float32) # 32 images of 240x420 shape
# Standard version. Takes 0.2s in average per epoch for the same processing.
for i in range(100):
new_data = [working_fn(image) for image in data]
# do further processing with the result
# Cached version. We store the result on memory.
def key_encode_fn(image: np.ndarray) -> str:
"""Return a cachable key for each image. We use a string of mean + std that should be unique enough."""
return f"{image.mean()}_{image.std()}"
cache = DictMemory(name="images", key_encode_fn=key_encode_fn)
# alternative, use: cache.map(working_fn, data)
for image in data:
cache[image] = working_fn(image)
for i in range(100):
new_data = [cache[image] for image in data]
# do further processing with the result
Decoator version
@cache_fn(NpyFS, key_encode_fn)
def working_fn(image: np.ndarray) -> np.ndarray:
"""Working function that takes on average 0.2s"""
if np.random.rand() <= 0.2:
time.sleep(1)
return (image - image.min()) / (image.max() - image.min())
def key_encode_fn(image: np.ndarray) -> str:
"""Return a cachable key for each image. We use a string of mean + std that should be unique enough."""
return f"{image.mean()}_{image.std()}"
for i in range(100):
new_data = working_fn(image)
# do further processing with the result
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Close
Hashes for python-simple-caching-0.3.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8c3e180ae28715d6f6f7f7e3712dde7163918164d4743b0d2c026950299ce970 |
|
MD5 | d293685228fca9c59771978a7adb96bd |
|
BLAKE2b-256 | 1e474204a81a4f4b1d188bd0e46e61245ed349190dd57bed2895488dcdb6a94b |