Skip to main content

Persistent compressed expiring dict in Python, backed up by sqlite3 and pickle, with auto-cleaning and auto-vacuuming semantics

Project description

expiringsqlitedict -- persistent ``dict``, backed-up by SQLite and pickle


.. _Downloads:
.. _License:

A lightweight wrapper around Python's sqlite3 database with a simple, Pythonic
dict-like interface. This fork is modified to implement a metatable and
automatic expiring and vacuuming semantics, as well as some appropriate locking.
This also compresses values automatically.

.. code-block:: python

>>> from expiringsqlitedict import SqliteDict
>>> with SqliteDict('./my_db.sqlite', autocommit=True) as mydict:
>>> mydict['some_key'] = any_picklable_object
>>> print mydict['some_key'] # prints the new value
>>> for key, value in mydict.iteritems():
>>> print key, value
>>> print len(mydict) # etc... all dict functions work

Pickle is used internally to (de)serialize the values. Keys are arbitrary strings,
values arbitrary pickle-able objects. This must be used within a context


* Values can be **any picklable objects** (uses ``cPickle`` with the highest protocol).
* Support for **access from multiple programs or threads**, using a lockfile.
* Support for **custom serialization or compression**:

.. code-block:: python

# use JSON instead of pickle
>>> import json
>>> mydict = SqliteDict('./my_db.sqlite', encode=json.dumps, decode=json.loads)

# apply zlib compression after pickling
>>> import zlib, pickle, sqlite3
>>> def my_encode(obj):
... return sqlite3.Binary(zlib.compress(pickle.dumps(obj, pickle.HIGHEST_PROTOCOL)))
>>> def my_decode(obj):
... return pickle.loads(zlib.decompress(bytes(obj)))
>>> mydict = SqliteDict('./my_db.sqlite', encode=my_encode, decode=my_decode)


The module has no dependencies beyond Python itself.

Install or upgrade with::

pip install expiringsqlitedict

or from the `source tar.gz <>`_::

python install


Standard Python document strings are inside the module:

.. code-block:: python

>>> import expiringsqlitedict
>>> help(expiringsqlitedict)

(but it's just ``dict`` with a commit, really).

**Beware**: because of Python semantics, ``expiringsqlitedict`` cannot know when
a mutable SqliteDict-backed entry was modified in RAM. For example,
``mydict.setdefault('new_key', []).append(1)`` will leave ``mydict['new_key']``
equal to empty list, not ``[1]``. You'll need to explicitly assign the mutated
object back to SqliteDict to achieve the same effect:

.. code-block:: python

>>> val = mydict.get('new_key', [])
>>> val.append(1) # sqlite DB not updated here!
>>> mydict['new_key'] = val # now updated

For developers


# pip install nose
# pip install coverage

To perform all tests::

# make test-all

To perform all tests with coverage::

# make test-all-with-coverage

Comments, bug reports

``expiringsqlitedict`` resides on `github <>`_. You can file issues or pull
requests there.


``expiringsqlitedict`` is open source software released under the
`Apache 2.0 license <>`_.
Copyright (c) 2011-2018 `Radim Řehůřek <>`_ and
contributors. The changes in this fork copyright (c) 2018 Absolute Performance,

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for expiringsqlitedict, version 1.2.3
Filename, size File type Python version Upload date Hashes
Filename, size expiringsqlitedict-1.2.3-py2.py3-none-any.whl (11.1 kB) File type Wheel Python version py2.py3 Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page