cs-cache 20241122

A few caching data structures and other lossy things with capped sizes.

A few caching data structures and other lossy things with capped sizes.

Latest release 20241122:

• convof, ConvCache.convof: new options force=False parameter to bypass existing cache entries.
• ConvCache.convof: check the conv_subpath using cs.fs.validate_rpath.

Class CachingMapping(cs.resources.MultiOpenMixin, collections.abc.MutableMapping)

A caching front end for another mapping. This is intended as a generic superclass for a proxy to a slower mapping such as a database or remote key value store.

Note that this subclasses MultiOpenMixin to start/stop the worker Thread. Users must enclose use of a CachingMapping in a with statement. If subclasses also subclass MultiOpenMixin their startup_shutdown method needs to also call our startup_shutdown method.

Example:

class Store:
  """ A key value store with a slower backend.
  """
  def __init__(self, mapping:Mapping):
    self.mapping = CachingMapping(mapping)

.....
S = Store(slow_mapping)
with S:
  ... work with S ...

CachingMapping.__init__(self, mapping: Mapping, *, max_size=1024, queue_length=1024, delitem_bg: Optional[Callable[[Any], cs.result.Result]] = None, setitem_bg: Optional[Callable[[Any, Any], cs.result.Result]] = None, missing_fallthrough: bool = False): Initialise the cache.

Parameters:

• mapping: the backing store, a mapping
• max_size: optional maximum size for the cache, default 1024
• queue_length: option size for the queue to the worker, default 1024
• delitem_bg: optional callable to queue a delete of a key in the backing store; if unset then deleted are serialised in the worker thread
• setitem_bg: optional callable to queue setting the value for a key in the backing store; if unset then deleted are serialised in the worker thread
• missing_fallthrough: is true (default False) always fall back to the backing mapping if a key is not in the cache

CachingMapping.flush(self): Wait for outstanding requests in the queue to complete. Return the UNIX time of completion.

CachingMapping.items(self): Generator yielding (k,v) pairs.

CachingMapping.keys(self): Generator yielding the keys.

Class ConvCache(cs.fs.HasFSPath)

A cache for conversions of file contents such as thumbnails or transcoded media, etc. This keeps cached results in a file tree based on a content key, whose default function is cs.hashutils.file_checksum('blake3').

ConvCache.__init__(self, fspath: Optional[str] = None, content_key_func=None): Initialise a ConvCache.

Parameters:

• fspath: optional base path of the cache, default from ConvCache.DEFAULT_CACHE_BASEPATH; if this does not exist it will be created using os.mkdir
• content_key_func: optional function to compute a key from the contents of a file, default cs.hashindex.file_checksum (the blake3 hash of the contents)

ConvCache.content_key(self, srcpath): Return a content key for the filesystem path srcpath.

ConvCache.content_subpath(self, srcpath) -> str: Return the content key based subpath component.

This default assumes the content key is a hash code and breaks it hex representation into a 3 level hierarchy such as 'd6/d9/c510785c468c9aa4b7bda343fb79'.

ConvCache.convof(self, srcpath, conv_subpath, conv_func, *, ext=None, force=False) -> str: Return the filesystem path of the cached conversion of srcpath via conv_func.

Parameters:

• srcpath: the source filesystem path
• conv_subpath: a name for the conversion which encompasses the salient aspaects such as 'png/64/64' for a 64x64 pixel thumbnail in PNG format
• conv_func: a callable of the form conv_func(srcpath,dstpath) to convert the contents of srcpath and write the result to the filesystem path dstpath
• ext: an optional filename extension, default from the first component of conv_subpath
• force: option flag to require conversion even if the cache has an entry

convof(srcpath, conv_subpath, conv_func, *, ext=None, force=False)

ConvCache.convof using the default cache.

Class LRU_Cache

A simple least recently used cache.

Unlike functools.lru_cache this provides on_add and on_remove callbacks.

LRU_Cache.__init__(self, max_size, *, on_add=None, on_remove=None): Initialise the LRU_Cache with maximum size max, additon callback on_add and removal callback on_remove.

LRU_Cache.__delitem__(self, key): Delete the specified key, calling the on_remove callback.

LRU_Cache.__setitem__(self, key, value): Store the item in the cache. Prune if necessary.

LRU_Cache.flush(self): Clear the cache.

LRU_Cache.get(self, key, default=None): Mapping method: get value for key or default.

LRU_Cache.items(self): Items from the cache.

LRU_Cache.keys(self): Keys from the cache.

lru_cache(max_size=None, cache=None, on_add=None, on_remove=None)

Enhanced workalike of @functools.lru_cache.

Release Log

Release 20241122:

• convof, ConvCache.convof: new options force=False parameter to bypass existing cache entries.
• ConvCache.convof: check the conv_subpath using cs.fs.validate_rpath.

Release 20240422.1: ConvCache docstring update.

Release 20240422: New ConvCache and convof: a cache for conversions of file contents such as thumbnails or transcoded media.

Release 20240412:

• New CachingMapping, a caching front end for another mapping.
• LRU_Cache: add keys() and items().

Release 20181228: Initial PyPI release.