Skip to main content

Persistent compressed expiring dict in Python, backed up by sqlite3 and pickle, with auto-cleaning and auto-vacuuming semantics

Project description

A lightweight wrapper around Python’s sqlite3 database with a simple, Pythonic dict-like interface. This fork is modified to implement a metatable and automatic expiring and vacuuming semantics, as well as some appropriate locking. This also compresses values automatically.

>>> from expiringsqlitedict import SqliteDict
>>> with SqliteDict('./my_db.sqlite', autocommit=True) as mydict:
>>>     mydict['some_key'] = any_picklable_object
>>>     print mydict['some_key']  # prints the new value
>>>     for key, value in mydict.iteritems():
>>>         print key, value
>>>     print len(mydict) # etc... all dict functions work

Pickle is used internally to (de)serialize the values. Keys are arbitrary strings, values arbitrary pickle-able objects. This must be used within a context manager.

Features

  • Values can be any picklable objects (uses cPickle with the highest protocol).

  • Support for access from multiple programs or threads, using a lockfile.

  • Support for custom serialization or compression:

    # use JSON instead of pickle
    >>> import json
    >>> mydict = SqliteDict('./my_db.sqlite', encode=json.dumps, decode=json.loads)
    
    # apply zlib compression after pickling
    >>> import zlib, pickle, sqlite3
    >>> def my_encode(obj):
    ...     return sqlite3.Binary(zlib.compress(pickle.dumps(obj, pickle.HIGHEST_PROTOCOL)))
    >>> def my_decode(obj):
    ...     return pickle.loads(zlib.decompress(bytes(obj)))
    >>> mydict = SqliteDict('./my_db.sqlite', encode=my_encode, decode=my_decode)
    

Installation

The module has no dependencies beyond Python itself.

Install or upgrade with:

pip install expiringsqlitedict

or from the source tar.gz:

python setup.py install

Documentation

Standard Python document strings are inside the module:

>>> import expiringsqlitedict
>>> help(expiringsqlitedict)

(but it’s just dict with a commit, really).

Beware: because of Python semantics, expiringsqlitedict cannot know when a mutable SqliteDict-backed entry was modified in RAM. For example, mydict.setdefault('new_key', []).append(1) will leave mydict['new_key'] equal to empty list, not [1]. You’ll need to explicitly assign the mutated object back to SqliteDict to achieve the same effect:

>>> val = mydict.get('new_key', [])
>>> val.append(1)  # sqlite DB not updated here!
>>> mydict['new_key'] = val  # now updated

For developers

Install:

# pip install nose
# pip install coverage

To perform all tests:

# make test-all

To perform all tests with coverage:

# make test-all-with-coverage

Comments, bug reports

expiringsqlitedict resides on github. You can file issues or pull requests there.


expiringsqlitedict is open source software released under the Apache 2.0 license. Copyright (c) 2011-2018 Radim Řehůřek and contributors. The changes in this fork copyright (c) 2018 Absolute Performance, Inc.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for expiringsqlitedict, version 1.2.4
Filename, size File type Python version Upload date Hashes
Filename, size expiringsqlitedict-1.2.4-py2.py3-none-any.whl (11.1 kB) File type Wheel Python version py2.py3 Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page