Skip to main content

table-oriented abstraction layer over key-value stores

Project description

kvlayer
=======

``kvlayer`` is a database abstraction layer providing a simple key-value store to applications. For development purposes this can run against an in-memory implementation or a server such as [Redis](http://redis.io/); in test and production, this can be switched to a relational database such as [PostgreSQL](http://postgresql.org/) or a cluster database such as [Accumulo](http://accumulo.apache.org/) or [Riak](http://basho.com/riak/).

Configuration
-------------

``kvlayer`` depends on the [Yakonfig](https://github.com/diffeo/yakonfig/) library to get its configuration information. Configuration is in a YAML file passed into the application. This includes a *storage type*, indicating which backend to use, and an *application name* and a *namespace*, both of which distinguish different applications sharing the same database.

```yaml
kvlayer:
storage_type: local # in-memory data store
app_name: kvlayer
namespace: kvlayer
```

kvlayer API
-----------

Applications see multiple kvlayer *tables*, which may be implemented as database-native tables for databases that have that concept. Each row has a key and a value. The keys are Python tuples, with some consistent set of types; tuple parts may be strings, integers, or UUIDs. Values are always Python byte strings.

There are four basic operations kvlayer makes available. ``put()`` writes one or more key-value pairs into the database. ``get()`` retrieves key-value pairs with known fixed keys. ``scan()`` retrieves key-value pairs within a range of keys. ``delete()`` removes specific keys.

A minimal kvlayer application would look like:

```python
import argparse
import kvlayer
import yakonfig

parser = argparse.ArgumentParser()
yakonfig.parse_args(parser, [yakonfig, kvlayer])
kvl = kvlayer.client()
kvl.setup_namespace({'table': (str,)})

# Write values
kvl.put('table', (('foo',), 'one'), (('bar',), 'two'))

# Retrieve values
for k,v in kvl.get('table', ('foo',)):
assert k == 'foo'
print v

# Scan values
for k,v in kvl.scan('table', (('a',), ('e',))):
print k
print v

# Scan keys
for k in kvl.scan_keys('table', (('e',), ('z',))):
print k

# Delete values
kvl.delete('table', ('foo',))
```

Other notes
-----------

See details of [testing on Accumulo using saltstack](accumulo-tests.md).

For throughput testing, see [kvlayer_throughput_tests](https://github.com/diffeo/kvlayer/blob/0.4.5/kvlayer/tests/test_throughput.py).

For example, using various single-node EC2 instances, random
reads/writes experiences these rates:

| num_workers | storage_type | read MB/sec | write MB/sec | |
|-------------|--------------|-------------|--------------|---|
| 100 | redis | 99.6 | 57.3 |m1.xlarge |
| 50 | redis | 93.7 | 56.5 |m1.xlarge |
| 25 | redis | 66.9 | 33.8 |m1.xlarge |
| 80 | postgres | 34.2 | 14.4 |m1.medium |
| 50 | postgres | 33.1 | 14.1 |m1.medium |
| 25 | postgres | 30.1 | 13.7 |m1.medium |
| 100 | accumulo | 17.2 | 13.6 |m1.large |
| 50 | accumulo | 21.9 | 16.0 |m1.large |
| 25 | accumulo | 24.7 | 16.6 |m1.large |

TODO: gather more stats.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

kvlayer-0.5.9.tar.gz (75.7 kB view hashes)

Uploaded source

Built Distribution

kvlayer-0.5.9-py2.7.egg (217.7 kB view hashes)

Uploaded 2 7

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page