This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).
Help us improve Python packaging - Donate today!

table-oriented abstraction layer over key-value stores

Project Description
kvlayer
=======

``kvlayer`` is a database abstraction layer providing a simple key-value store to applications. For development purposes this can run against an in-memory implementation or a server such as [Redis](http://redis.io/); in test and production, this can be switched to a relational database such as [PostgreSQL](http://postgresql.org/) or a cluster database such as [Accumulo](http://accumulo.apache.org/) or [Riak](http://basho.com/riak/).

Configuration
-------------

``kvlayer`` depends on the [Yakonfig](https://github.com/diffeo/yakonfig/) library to get its configuration information. Configuration is in a YAML file passed into the application. This includes a *storage type*, indicating which backend to use, and an *application name* and a *namespace*, both of which distinguish different applications sharing the same database.

```yaml
kvlayer:
storage_type: local # in-memory data store
app_name: kvlayer
namespace: kvlayer
```

kvlayer API
-----------

Applications see multiple kvlayer *tables*, which may be implemented as database-native tables for databases that have that concept. Each row has a key and a value. The keys are Python tuples, with some consistent set of types; tuple parts may be strings, integers, or UUIDs. Values are always Python byte strings.

There are four basic operations kvlayer makes available. ``put()`` writes one or more key-value pairs into the database. ``get()`` retrieves key-value pairs with known fixed keys. ``scan()`` retrieves key-value pairs within a range of keys. ``delete()`` removes specific keys.

A minimal kvlayer application would look like:

```python
import argparse
import kvlayer
import yakonfig

parser = argparse.ArgumentParser()
yakonfig.parse_args(parser, [yakonfig, kvlayer])
kvl = kvlayer.client()
kvl.setup_namespace({'table': (str,)})

# Write values
kvl.put('table', (('foo',), 'one'), (('bar',), 'two'))

# Retrieve values
for k,v in kvl.get('table', ('foo',)):
assert k == 'foo'
print v

# Scan values
for k,v in kvl.scan('table', (('a',), ('e',))):
print k
print v

# Scan keys
for k in kvl.scan_keys('table', (('e',), ('z',))):
print k

# Delete values
kvl.delete('table', ('foo',))
```

Other notes
-----------

See details of [testing on Accumulo using saltstack](accumulo-tests.md).

For throughput testing, see [kvlayer_throughput_tests](https://github.com/diffeo/kvlayer/blob/0.4.5/kvlayer/tests/test_throughput.py).

For example, using various single-node EC2 instances, random
reads/writes experiences these rates:

| num_workers | storage_type | read MB/sec | write MB/sec | |
|-------------|--------------|-------------|--------------|---|
| 100 | redis | 99.6 | 57.3 |m1.xlarge |
| 50 | redis | 93.7 | 56.5 |m1.xlarge |
| 25 | redis | 66.9 | 33.8 |m1.xlarge |
| 80 | postgres | 34.2 | 14.4 |m1.medium |
| 50 | postgres | 33.1 | 14.1 |m1.medium |
| 25 | postgres | 30.1 | 13.7 |m1.medium |
| 100 | accumulo | 17.2 | 13.6 |m1.large |
| 50 | accumulo | 21.9 | 16.0 |m1.large |
| 25 | accumulo | 24.7 | 16.6 |m1.large |

TODO: gather more stats.
Release History

Release History

This version
History Node

0.5.9

History Node

0.5.9.dev2

History Node

0.5.8

History Node

0.5.8.dev1

History Node

0.5.7

History Node

0.5.7.dev1

History Node

0.5.6

History Node

0.5.6.dev1

History Node

0.5.5

History Node

0.5.5.dev2

History Node

0.5.4

History Node

0.5.4.dev4

History Node

0.5.4.dev3

History Node

0.5.3

History Node

0.5.3.dev9

History Node

0.5.3.dev8

History Node

0.5.3.dev7

History Node

0.5.3.dev6

History Node

0.5.3.dev5

History Node

0.5.3.dev2

History Node

0.5.3.dev1

History Node

0.5.2

History Node

0.5.2.dev1

History Node

0.5.1

History Node

0.5.0

History Node

0.4.21

History Node

0.4.21.dev4

History Node

0.4.20

History Node

0.4.19

History Node

0.4.18

History Node

0.4.17

History Node

0.4.16

History Node

0.4.16.dev4

History Node

0.4.16.dev3

History Node

0.4.16.dev2

History Node

0.4.15

History Node

0.4.15.dev5

History Node

0.4.15.dev4

History Node

0.4.15.dev3

History Node

0.4.15.dev2

History Node

0.4.15.dev1

History Node

0.4.14

History Node

0.4.14.dev1

History Node

0.4.13

History Node

0.4.12

History Node

0.4.12.dev11

History Node

0.4.12.dev10

History Node

0.4.12.dev9

History Node

0.4.12.dev6

History Node

0.4.12.dev4

History Node

0.4.12.dev2

History Node

0.4.12.dev1

History Node

0.4.11

History Node

0.4.11.dev2

History Node

0.4.11.dev1

History Node

0.4.10

History Node

0.4.10.dev2

History Node

0.4.10.dev1

History Node

0.4.9

History Node

0.4.8.dev23

History Node

0.4.8.dev20

History Node

0.4.8.dev17

History Node

0.4.8.dev16

History Node

0.4.8.dev15

History Node

0.4.8.dev14

History Node

0.4.8.dev13

History Node

0.4.8.dev8

History Node

0.4.8.dev7

History Node

0.4.8.dev5

History Node

0.4.8.dev2

History Node

0.4.8.dev1

History Node

0.4.7.dev6

History Node

0.4.7.dev3

History Node

0.4.6

History Node

0.4.6.dev3

History Node

0.4.5

History Node

0.4.2.dev1

History Node

0.4.1

History Node

0.3.2

History Node

0.2.20.dev2

History Node

0.2.16

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
kvlayer-0.5.9-py2.7.egg (217.7 kB) Copy SHA256 Checksum SHA256 2.7 Egg Jul 31, 2015
kvlayer-0.5.9.tar.gz (75.7 kB) Copy SHA256 Checksum SHA256 Source Jul 31, 2015

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting