Elevator 0.5

On disk key/value store based on levelDB backend

Key-Value store written in Python and based on levelDB, allows high performance on-disk bulk read/write.

Allows async, multithreaded and/or remote access to a multi-leveldb backend.

Relying on the zeromq network library and msgpack serialization format, it is made to be portable between languages and platforms.

See Documentation for more details

Depends on

• zmq-3.X

• pyzmq (built with zmq-3.X)

• leveldb

• plyvel

Installation

The easy way:

pip install elevator

The hacker way:

$ pip install fabric
$ fab build
$ pip install -r requirements.txt
$ python setup.py install

Usage

When elevator is installed, you can then launch the server using the elevator executable. Note that a –daemon option is disposable, and allows you to run elevator server as a daemon, storing it’s pid in .pid file in /tmp.

See config/elevator.conf for an example of Elevator configuration.

Example:

$ elevator --help
usage: elevator -h

Elevator command line manager

optional arguments:
    -h, --help       show this help message and exit
    -d, --daemon      Launch elevator as a daemon
    -c, --config      Path to elevator server config file, eventually
    -t, --transport   Transfert protocol (tcp | ipc)
    -b, --bind        Ip to bind server to
    -p, --port        Port the server should listen on
    -w, --workers     How many workers should be spawned (Threads with concurrent access to all the db store)
    -P, --paranoid    If option is set, Elevator will shutdown and log on first unhandled exception

Clients

Python: py-elevator

Go: go-elevator (Early early, so early version)

Clojure : Coming soon

C : Coming soon

Thanks

Thanks to srinikom for its leveldb-server which was a very good base to start from.

n.n.n / 2013-02-01

• Remove: legacy setup_loggers function

• fix #123: exposing a database object

• ref #123: Renamed DatabasesHandler to DatabaseStore

• Fix: elevator benchmarks

• Fix: supervisor test should remove their tests files

• Add: tests for backend atm

• Add: backend supervisor tests + fixes

• update: enhance backend majordome management

• fix #125: backend does not instantiate it’s own DatabasesHandler anymore

• Fix: elevator tests fakers now uses a clear files/dirs pattern

• Update: more obvious DatabaseHandler args names

• Add: benchmarks using hurdles and pyelevator

• update #120 : Auto re-mount unmounted database on new requests

• Fix : backend properly tears down workers

• fix #120, fix #91: Implement Majordom watcher thread

• Update #120: set databases last access marker

• Update 120: move ocd worker to backend module

• Update #121: implement last activity action on workers

• Update #121: Documented worker

• Update #121: Workers poll to reduce cpu usage + backend refactoring

• Update #121: use an internal message protocol between supervisor and workers

• Update: Moved the backend elements in their own module

• Fix #122: workers now set their processing state

• Refactor: moved loggers init in their own log module

• Update #121: fixed workers stop action

• Update #121: Added constants to normalize interaction with workers

• Add #121: basic workers supervisor implementation, implies a lot of refactoring

• Update: rename server poller

• Update: use ROUTER/DEALER terminology and rename workerpool and proxy to backend and frontend

• Update: renamed conf module to args

0.4b / 2013-01-28

• Fix: Refactor api tests

• Fix #119: Range and Slice now support include_key, and include_value params

• Remove: max cache management + Add: Lru cache and bloom filters

0.4a / 2013-01-22

• Add : Implement PING command

• Add : Cli module

• Add : Debian packaging files

• Update: Use plyvel leveldb backend

• Update: Use plyvel bloom filter in read operations

• Update: Add experimental command line doc

• Update: Set fabfile as a module

• Update: Documentation to fit with plyvel

• Update #114: Run MGet against db snapshot

• Update : working cmdline

• Fix #114: Enhance MGET perfs by acting on a min/max keys range slice

• Fix #113: handle MGET arguments in command line

• Many other little updates and fixes, see logs

0.4 / 2012-10-22

• Add: restore theme

• Add : Base sphinx documentation

• Update : new License MIT

• Fix #86: IOError when bad config file supplied as cmdline argument

• Fix #95: Elevator will start and log errors even though some databases are corrupted

• Fix : log-level debug messages format

• Fix : travis, tests, requirements

0.3d / 2012-10-19

• Add : Request error for invalid request messages

• Update #91: Mount default at server startup

• Update #91: Mount/Unmount command + auto-mount on connect

• Update #91: add a ticker class, which executes a function every x seconds

• Update #30, Update #99: Compress Responses on demande (request.meta[‘compression’])

• Update #88, Update #99: now responses comes in two parts header+content

• Update #88: Fix MGet, Range, Slice return values types to suite with new responses format

• Update #88: Refactored Request/Responses format

• Update : Refactored DatabasesHandler internal storage

• Update : Few refactoring on loggers handling

• Update : Refactored DBConnect no more db_uid to provide in request

• Fix #97: provide mono-letters args

• Fix #89: Log requests/responses on log-level=DEBUG

• Fix #87: Refactored logging

• Fix #100: Non blocking workers, graceful shutdown == PERFORMANCES

• Fix #98: Activity logging on both file and stdout

• Fix #101: fixed ipc handling

• Fix : api tests for compatibility with new Req/Resp

• Fix : refactored tests for new Range/Slice behavior when size == 1

• Fix : Mount/Unmount passed args