Skip to main content

LeakDB is a very simple and fast key value store for Python

Project description

Why ?

For the fun o/

Overview

LeakDB is a very simple and fast key value store for Python.

All data is stored in memory and the persistence is defined by the user. A max queue size can be defined for a auto-flush.

API

>>> from leakdb import PersistentQueueStorage
>>> leak = PersistentQueueStorage(filename='/tmp/foobar.db')

# set the value of a key
>>> leak.set('bar', {'foo': 'bar'})
>>> leak.set('foo', 2, key_prefix='bar_')

# increment a key
>>> leak.incr(key='bar_foo', delta=5)
 7

>>> leak.incr(key='foobar', initial_value=1000)
 1000

# looks up multiple keys
>>> leak.get_multi(keys=['bar', 'foobar'])
 {u'foobar': 1000, u'bar': {u'foo': u'bar'}}

# ensure changes are sent to disk
>>> print leak
 /tmp/foobar.db 12288 bytes :: 3 items in queue :: 3 items in storage memory

>>> leak.flush(force=True)
 /tmp/foobar.db 12338 bytes :: 0 items in queue :: 3 items in storage memory

>>> leak.close()

STORAGE

  • DefaultStorage :: The default storage, all API operations are implemented set set_multi incr decr get_multi delete
  • QueueStorage :: Use the DefaultStorage with a queue. You can override the QueueStorage.worker_process method and make what you want when the flush method is called.
from leakdb import QueueStorage

class MyQueueStorage(QueueStorage):

    def worker_process(self, item):
        """ Default action execute by each worker.
            Must return a True statement to remove the item,
            otherwise the worker put the item into the queue.
        """
        logger.info('process item :: {}'.format(item))
        return True
  • PersistentStorage :: Use the DefaultStorage, otherwise each operation is stored through the shelve module.
  • PersistentQueueStorage :: Use the QueueStorage and the PersistentStorage.
# see also the API part
from leakdb import PersistentQueueStorage

storage = PersistentQueueStorage(filename="/tmp/foobar.db",  maxsize=1, workers=1)
# the queue is auto-flush, each operations check the queue size
storage.set('foo', 1)

TODO

  • finish the transport layer through zeroMQ
  • cleanup the code
  • write the unittests
  • write a CLI
  • benchmark each storage

Project details


Release history Release notifications

History Node

0.2

This version
History Node

0.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
LeakDB-0.1.tar.gz (5.0 kB) Copy SHA256 hash SHA256 Source None Aug 1, 2014

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page