kvlayer
=======
``kvlayer`` is a database abstraction layer providing a simple key-value store to applications. For development purposes this can run against an in-memory implementation or a server such as [Redis](http://redis.io/); in test and production, this can be switched to a relational database such as [PostgreSQL](http://postgresql.org/) or a cluster database such as [Accumulo](http://accumulo.apache.org/) or [Riak](http://basho.com/riak/).
Configuration
-------------
``kvlayer`` depends on the [Yakonfig](https://github.com/diffeo/yakonfig/) library to get its configuration information. Configuration is in a YAML file passed into the application. This includes a *storage type*, indicating which backend to use, and an *application name* and a *namespace*, both of which distinguish different applications sharing the same database.
```yaml
kvlayer:
storage_type: local # in-memory data store
app_name: kvlayer
namespace: kvlayer
```
kvlayer API
-----------
Applications see multiple kvlayer *tables*, which may be implemented as database-native tables for databases that have that concept. Each row has a key and a value. The keys are Python tuples, with some consistent set of types; tuple parts may be strings, integers, or UUIDs. Values are always Python byte strings.
There are four basic operations kvlayer makes available. ``put()`` writes one or more key-value pairs into the database. ``get()`` retrieves key-value pairs with known fixed keys. ``scan()`` retrieves key-value pairs within a range of keys. ``delete()`` removes specific keys.
A minimal kvlayer application would look like:
```python
import argparse
import kvlayer
import yakonfig
parser = argparse.ArgumentParser()
yakonfig.parse_args(parser, [yakonfig, kvlayer])
kvl = kvlayer.client()
kvl.setup_namespace({'table': (str,)})
# Write values
kvl.put('table', (('foo',), 'one'), (('bar',), 'two'))
# Retrieve values
for k,v in kvl.get('table', ('foo',)):
assert k == 'foo'
print v
# Scan values
for k,v in kvl.scan('table', (('a',), ('e',))):
print k
print v
# Scan keys
for k in kvl.scan_keys('table', (('e',), ('z',))):
print k
# Delete values
kvl.delete('table', ('foo',))
```
Other notes
-----------
See details of [testing on Accumulo using saltstack](accumulo-tests.md).
For throughput testing, see [kvlayer_throughput_tests](https://github.com/diffeo/kvlayer/blob/0.4.5/kvlayer/tests/test_throughput.py).
For example, using various single-node EC2 instances, random
reads/writes experiences these rates:
| num_workers | storage_type | read MB/sec | write MB/sec | |
|-------------|--------------|-------------|--------------|---|
| 100 | redis | 99.6 | 57.3 |m1.xlarge |
| 50 | redis | 93.7 | 56.5 |m1.xlarge |
| 25 | redis | 66.9 | 33.8 |m1.xlarge |
| 80 | postgres | 34.2 | 14.4 |m1.medium |
| 50 | postgres | 33.1 | 14.1 |m1.medium |
| 25 | postgres | 30.1 | 13.7 |m1.medium |
| 100 | accumulo | 17.2 | 13.6 |m1.large |
| 50 | accumulo | 21.9 | 16.0 |m1.large |
| 25 | accumulo | 24.7 | 16.6 |m1.large |
TODO: gather more stats.
=======
``kvlayer`` is a database abstraction layer providing a simple key-value store to applications. For development purposes this can run against an in-memory implementation or a server such as [Redis](http://redis.io/); in test and production, this can be switched to a relational database such as [PostgreSQL](http://postgresql.org/) or a cluster database such as [Accumulo](http://accumulo.apache.org/) or [Riak](http://basho.com/riak/).
Configuration
-------------
``kvlayer`` depends on the [Yakonfig](https://github.com/diffeo/yakonfig/) library to get its configuration information. Configuration is in a YAML file passed into the application. This includes a *storage type*, indicating which backend to use, and an *application name* and a *namespace*, both of which distinguish different applications sharing the same database.
```yaml
kvlayer:
storage_type: local # in-memory data store
app_name: kvlayer
namespace: kvlayer
```
kvlayer API
-----------
Applications see multiple kvlayer *tables*, which may be implemented as database-native tables for databases that have that concept. Each row has a key and a value. The keys are Python tuples, with some consistent set of types; tuple parts may be strings, integers, or UUIDs. Values are always Python byte strings.
There are four basic operations kvlayer makes available. ``put()`` writes one or more key-value pairs into the database. ``get()`` retrieves key-value pairs with known fixed keys. ``scan()`` retrieves key-value pairs within a range of keys. ``delete()`` removes specific keys.
A minimal kvlayer application would look like:
```python
import argparse
import kvlayer
import yakonfig
parser = argparse.ArgumentParser()
yakonfig.parse_args(parser, [yakonfig, kvlayer])
kvl = kvlayer.client()
kvl.setup_namespace({'table': (str,)})
# Write values
kvl.put('table', (('foo',), 'one'), (('bar',), 'two'))
# Retrieve values
for k,v in kvl.get('table', ('foo',)):
assert k == 'foo'
print v
# Scan values
for k,v in kvl.scan('table', (('a',), ('e',))):
print k
print v
# Scan keys
for k in kvl.scan_keys('table', (('e',), ('z',))):
print k
# Delete values
kvl.delete('table', ('foo',))
```
Other notes
-----------
See details of [testing on Accumulo using saltstack](accumulo-tests.md).
For throughput testing, see [kvlayer_throughput_tests](https://github.com/diffeo/kvlayer/blob/0.4.5/kvlayer/tests/test_throughput.py).
For example, using various single-node EC2 instances, random
reads/writes experiences these rates:
| num_workers | storage_type | read MB/sec | write MB/sec | |
|-------------|--------------|-------------|--------------|---|
| 100 | redis | 99.6 | 57.3 |m1.xlarge |
| 50 | redis | 93.7 | 56.5 |m1.xlarge |
| 25 | redis | 66.9 | 33.8 |m1.xlarge |
| 80 | postgres | 34.2 | 14.4 |m1.medium |
| 50 | postgres | 33.1 | 14.1 |m1.medium |
| 25 | postgres | 30.1 | 13.7 |m1.medium |
| 100 | accumulo | 17.2 | 13.6 |m1.large |
| 50 | accumulo | 21.9 | 16.0 |m1.large |
| 25 | accumulo | 24.7 | 16.6 |m1.large |
TODO: gather more stats.
2 maintainers
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
kvlayer-0.5.9.tar.gz
(75.7 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
kvlayer-0.5.9-py2.7.egg
(217.7 kB
view details)
File details
Details for the file kvlayer-0.5.9.tar.gz.
File metadata
- Download URL: kvlayer-0.5.9.tar.gz
- Upload date:
- Size: 75.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
10981f685bdf120a9ad58068158b7d3038f21361954969ff63b4784a2a119257
|
|
| MD5 |
5baca2a9100e97bd0a7583222b35f14c
|
|
| BLAKE2b-256 |
dd0fca886387356fbe8c2cf6a3b67a824cf8ff3fdd578b365ec6e07778b9fc25
|
File details
Details for the file kvlayer-0.5.9-py2.7.egg.
File metadata
- Download URL: kvlayer-0.5.9-py2.7.egg
- Upload date:
- Size: 217.7 kB
- Tags: Egg
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
d1b44a07659e0a9a2b4c58a0be566d93120d345e8dbe3b26b4fae3096e1c3881
|
|
| MD5 |
3ae694070320782c0c11e04e7ccf37bc
|
|
| BLAKE2b-256 |
593aedacc324ce0f4b0343332d5d0f493b41fb261fd99785dfbe8a4b12220d02
|
