Persistent compressed expiring dict in Python, backed up by sqlite3 and pickle, with auto-cleaning and auto-vacuuming semantics
Project description
A lightweight wrapper around Python’s sqlite3 database with a simple, Pythonic dict-like interface. This fork is modified to implement a metatable and automatic expiring and vacuuming semantics, as well as some appropriate locking. This also compresses values automatically.
>>> from expiringsqlitedict import SqliteDict
>>> with SqliteDict('./my_db.sqlite', autocommit=True) as mydict:
>>> mydict['some_key'] = any_picklable_object
>>> print mydict['some_key'] # prints the new value
>>> for key, value in mydict.iteritems():
>>> print key, value
>>> print len(mydict) # etc... all dict functions work
Pickle is used internally to (de)serialize the values. Keys are arbitrary strings, values arbitrary pickle-able objects. This must be used within a context manager.
Features
Values can be any picklable objects (uses cPickle with the highest protocol).
Support for access from multiple programs or threads, using a lockfile.
Support for custom serialization or compression:
# use JSON instead of pickle >>> import json >>> mydict = SqliteDict('./my_db.sqlite', encode=json.dumps, decode=json.loads) # apply zlib compression after pickling >>> import zlib, pickle, sqlite3 >>> def my_encode(obj): ... return sqlite3.Binary(zlib.compress(pickle.dumps(obj, pickle.HIGHEST_PROTOCOL))) >>> def my_decode(obj): ... return pickle.loads(zlib.decompress(bytes(obj))) >>> mydict = SqliteDict('./my_db.sqlite', encode=my_encode, decode=my_decode)
Installation
The module has no dependencies beyond Python itself.
Install or upgrade with:
pip install expiringsqlitedict
or from the source tar.gz:
python setup.py install
Documentation
Standard Python document strings are inside the module:
>>> import expiringsqlitedict
>>> help(expiringsqlitedict)
(but it’s just dict with a commit, really).
Beware: because of Python semantics, expiringsqlitedict cannot know when a mutable SqliteDict-backed entry was modified in RAM. For example, mydict.setdefault('new_key', []).append(1) will leave mydict['new_key'] equal to empty list, not [1]. You’ll need to explicitly assign the mutated object back to SqliteDict to achieve the same effect:
>>> val = mydict.get('new_key', [])
>>> val.append(1) # sqlite DB not updated here!
>>> mydict['new_key'] = val # now updated
For developers
Install:
# pip install nose # pip install coverage
To perform all tests:
# make test-all
To perform all tests with coverage:
# make test-all-with-coverage
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for expiringsqlitedict-1.2.4-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 17a6f905ae3d0e91c2f5be0dae875d9599f1d630f38ec483c7dcc90bf469c9fc |
|
MD5 | d6a9eb8cbd8dc4127726f21d647e6f92 |
|
BLAKE2b-256 | a7ea67e0a3d3a2b29804c92a1b749f9d691e9c40193a79d7b276060dc296e7d2 |
Comments, bug reports
expiringsqlitedict resides on github. You can file issues or pull requests there.
expiringsqlitedict is open source software released under the Apache 2.0 license. Copyright (c) 2011-2018 Radim Řehůřek and contributors. The changes in this fork copyright (c) 2018 Absolute Performance, Inc.