kvlayer
=======
``kvlayer`` is a database abstraction layer providing a simple key-value store to applications. For development purposes this can run against an in-memory implementation or a server such as [Redis](http://redis.io/); in test and production, this can be switched to a relational database such as [PostgreSQL](http://postgresql.org/) or a cluster database such as [Accumulo](http://accumulo.apache.org/) or [Riak](http://basho.com/riak/).
Configuration
-------------
``kvlayer`` depends on the [Yakonfig](https://github.com/diffeo/yakonfig/) library to get its configuration information. Configuration is in a YAML file passed into the application. This includes a *storage type*, indicating which backend to use, and an *application name* and a *namespace*, both of which distinguish different applications sharing the same database.
```yaml
kvlayer:
storage_type: local # in-memory data store
app_name: kvlayer
namespace: kvlayer
```
kvlayer API
-----------
Applications see multiple kvlayer *tables*, which may be implemented as database-native tables for databases that have that concept. Each row has a key and a value. The keys are Python tuples, with some consistent set of types; tuple parts may be strings, integers, or UUIDs. Values are always Python byte strings.
There are four basic operations kvlayer makes available. ``put()`` writes one or more key-value pairs into the database. ``get()`` retrieves key-value pairs with known fixed keys. ``scan()`` retrieves key-value pairs within a range of keys. ``delete()`` removes specific keys.
A minimal kvlayer application would look like:
```python
import argparse
import kvlayer
import yakonfig
parser = argparse.ArgumentParser()
yakonfig.parse_args(parser, [yakonfig, kvlayer])
kvl = kvlayer.client()
kvl.setup_namespace({'table': (str,)})
# Write values
kvl.put('table', (('foo',), 'one'), (('bar',), 'two'))
# Retrieve values
for k,v in kvl.get('table', ('foo',)):
assert k == 'foo'
print v
# Scan values
for k,v in kvl.scan('table', (('a',), ('e',))):
print k
print v
# Scan keys
for k in kvl.scan_keys('table', (('e',), ('z',))):
print k
# Delete values
kvl.delete('table', ('foo',))
```
Other notes
-----------
See details of [testing on Accumulo using saltstack](accumulo-tests.md).
For throughput testing, see [kvlayer_throughput_tests](https://github.com/diffeo/kvlayer/blob/0.4.5/kvlayer/tests/test_throughput.py).
For example, using various single-node EC2 instances, random
reads/writes experiences these rates:
| num_workers | storage_type | read MB/sec | write MB/sec | |
|-------------|--------------|-------------|--------------|---|
| 100 | redis | 99.6 | 57.3 |m1.xlarge |
| 50 | redis | 93.7 | 56.5 |m1.xlarge |
| 25 | redis | 66.9 | 33.8 |m1.xlarge |
| 80 | postgres | 34.2 | 14.4 |m1.medium |
| 50 | postgres | 33.1 | 14.1 |m1.medium |
| 25 | postgres | 30.1 | 13.7 |m1.medium |
| 100 | accumulo | 17.2 | 13.6 |m1.large |
| 50 | accumulo | 21.9 | 16.0 |m1.large |
| 25 | accumulo | 24.7 | 16.6 |m1.large |
TODO: gather more stats.
=======
``kvlayer`` is a database abstraction layer providing a simple key-value store to applications. For development purposes this can run against an in-memory implementation or a server such as [Redis](http://redis.io/); in test and production, this can be switched to a relational database such as [PostgreSQL](http://postgresql.org/) or a cluster database such as [Accumulo](http://accumulo.apache.org/) or [Riak](http://basho.com/riak/).
Configuration
-------------
``kvlayer`` depends on the [Yakonfig](https://github.com/diffeo/yakonfig/) library to get its configuration information. Configuration is in a YAML file passed into the application. This includes a *storage type*, indicating which backend to use, and an *application name* and a *namespace*, both of which distinguish different applications sharing the same database.
```yaml
kvlayer:
storage_type: local # in-memory data store
app_name: kvlayer
namespace: kvlayer
```
kvlayer API
-----------
Applications see multiple kvlayer *tables*, which may be implemented as database-native tables for databases that have that concept. Each row has a key and a value. The keys are Python tuples, with some consistent set of types; tuple parts may be strings, integers, or UUIDs. Values are always Python byte strings.
There are four basic operations kvlayer makes available. ``put()`` writes one or more key-value pairs into the database. ``get()`` retrieves key-value pairs with known fixed keys. ``scan()`` retrieves key-value pairs within a range of keys. ``delete()`` removes specific keys.
A minimal kvlayer application would look like:
```python
import argparse
import kvlayer
import yakonfig
parser = argparse.ArgumentParser()
yakonfig.parse_args(parser, [yakonfig, kvlayer])
kvl = kvlayer.client()
kvl.setup_namespace({'table': (str,)})
# Write values
kvl.put('table', (('foo',), 'one'), (('bar',), 'two'))
# Retrieve values
for k,v in kvl.get('table', ('foo',)):
assert k == 'foo'
print v
# Scan values
for k,v in kvl.scan('table', (('a',), ('e',))):
print k
print v
# Scan keys
for k in kvl.scan_keys('table', (('e',), ('z',))):
print k
# Delete values
kvl.delete('table', ('foo',))
```
Other notes
-----------
See details of [testing on Accumulo using saltstack](accumulo-tests.md).
For throughput testing, see [kvlayer_throughput_tests](https://github.com/diffeo/kvlayer/blob/0.4.5/kvlayer/tests/test_throughput.py).
For example, using various single-node EC2 instances, random
reads/writes experiences these rates:
| num_workers | storage_type | read MB/sec | write MB/sec | |
|-------------|--------------|-------------|--------------|---|
| 100 | redis | 99.6 | 57.3 |m1.xlarge |
| 50 | redis | 93.7 | 56.5 |m1.xlarge |
| 25 | redis | 66.9 | 33.8 |m1.xlarge |
| 80 | postgres | 34.2 | 14.4 |m1.medium |
| 50 | postgres | 33.1 | 14.1 |m1.medium |
| 25 | postgres | 30.1 | 13.7 |m1.medium |
| 100 | accumulo | 17.2 | 13.6 |m1.large |
| 50 | accumulo | 21.9 | 16.0 |m1.large |
| 25 | accumulo | 24.7 | 16.6 |m1.large |
TODO: gather more stats.
2 maintainers
Release files for kvlayer 0.5.9
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| kvlayer-0.5.9.tar.gz | 75.7 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| kvlayer-0.5.9-py2.7.egg | Legacy Egg format | - | - | Details |
Total release size: 293.4 kB
Release files / kvlayer-0.5.9.tar.gz
| Download URL | kvlayer-0.5.9.tar.gz |
|---|---|
| Size | 75.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
10981f685bdf120a9ad58068158b7d3038f21361954969ff63b4784a2a119257
|
|
BLAKE2b-256 checksum How to use checksums |
dd0fca886387356fbe8c2cf6a3b67a824cf8ff3fdd578b365ec6e07778b9fc25
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Release files / kvlayer-0.5.9-py2.7.egg
| Download URL | kvlayer-0.5.9-py2.7.egg |
|---|---|
| Size | 217.7 kB |
| Tags | Egg |
|
SHA-256 checksum How to use checksums |
d1b44a07659e0a9a2b4c58a0be566d93120d345e8dbe3b26b4fae3096e1c3881
|
|
BLAKE2b-256 checksum How to use checksums |
593aedacc324ce0f4b0343332d5d0f493b41fb261fd99785dfbe8a4b12220d02
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
