sonya 0.6.6

About sonya:

• Written in Cython for speed and low-overhead

• Clean, memorable APIs

• Extensive support for Sophia’s features

• Python 2 and Python 3 support

• No 3rd-party dependencies besides Cython (for python>3)

About Sophia:

• Document store

• ACID transactions

• MVCC, optimistic, non-blocking concurrency with multiple readers and writers.

• Multiple databases per environment

• Multiple- and single-statement transactions across databases

• Prefix searches

• Automatic garbage collection

• Hot backup

• Compression

• Multi-threaded compaction

• mmap support, direct I/O support

• APIs for variety of statistics on storage engine internals

• BSD licensed

Some ideas of where Sophia might be a good fit:

• Running on application servers, low-latency / high-throughput

• Time-series

• Analytics / Events / Logging

• Full-text search

• Secondary-index for external data-store

Limitations:

• Not tested on Windows.

If you encounter any bugs in the library, please open an issue, including a description of the bug and any related traceback.

$ pip install Cython   # Optional
$ pip install sonya

$ pip install Cython   # Required
$ pip install git+https://github.com/mosquito/sonya#egg=sonya

$ pip install Cython
$ git clone https://github.com/mosquito/sonya
$ cd sonya
$ python setup.py build
$ python setup.py install

$ pip install pytest
$ pytest tests

from sonya import Environment, fields, Schema


class DictSchema(Schema):
    key = fields.PickleField(index=0)
    value = fields.PickleField()


env = Environment('/tmp/test-env')
db = env.database('test-database', DictSchema(), compression='zstd')
env.open()

document = db.document(key='foo', value=[1, 2, 3, 'bar'])

# Insert a document
db.set(document)

print(db.get(key='foo'))
# {'value': [1, 2, 3, 'bar'], 'key': 'foo'}

CRUD operations

Sonya

from sonya import Environment, fields, Schema


class DictSchema(Schema):
    key = fields.PickleField(index=0)
    value = fields.PickleField()


env = Environment('/tmp/test-env')
db = env.database('test-database', DictSchema(), compression='zstd')
env.open()

document = db.document(key='foo', value=[1, 2, 3, 'bar'])

# Create a document
db.set(document)

# Read document
document = db.get(key='foo')

# Update the document
document = db.document(key='foo', value=None)
db.set(document)

# Delete the document
document = db.document(key='foo', value=None)
db.delete(key='foo')

# Iterate through the database
for document in db.cursor():
   print(document)

# Delete multiple documents
# fastest method for remove multiple documents from database
db.delete_many(order='>=')

Fetching ranges (Cursors)

Because Sophia is an ordered data-store, performing ordered range scans is efficient. To retrieve a range of key-value pairs with Sonya dictionary lookup with a slice instead.

For finer-grained control over iteration, or to do prefix-matching, Sonya provides a cursor interface.

The cursor() method accepts special keyword parameter order and all key fields:

• order (default=`>=`) – semantics for matching the start key and ordering results.

from sonya import Environment, fields, Schema


class IntSchema(Schema):
    key = fields.UInt32Field(index=0)
    value = fields.PickleField()


env = Environment('/tmp/test-env')
db = env.database('test-integer-db', IntSchema(), compression='zstd')
env.open()


with db.transaction() as tx:
    for i in range(10000):
        tx.set(db.document(key=i, value=None))

# Iterate through the database
for document in db.cursor(order='>=', key=9995):
    print(document)

# {'key': 9995, 'value': None}
# {'key': 9996, 'value': None}
# {'key': 9997, 'value': None}
# {'key': 9998, 'value': None}
# {'key': 9999, 'value': None}

For prefix search use a part of the key and order:

from sonya import Environment, fields, Schema


class StringSchema(Schema):
    key = fields.StringField(index=0)
    value = fields.PickleField()


env = Environment('/tmp/test-env')
db = env.database('test-string-db', IntSchema(), compression='zstd')
env.open()


with db.transaction() as tx:
    for i in range(10000):
        tx.set(db.document(key=str(i), value=None))

# Iterate through the database
for document in db.cursor(order='>=', key='999'):
    print(document)

# {'value': None, 'key': '999'}
# {'value': None, 'key': '9990'}
# {'value': None, 'key': '9991'}
# {'value': None, 'key': '9992'}
# {'value': None, 'key': '9993'}
# {'value': None, 'key': '9994'}
# {'value': None, 'key': '9995'}
# {'value': None, 'key': '9996'}
# {'value': None, 'key': '9997'}
# {'value': None, 'key': '9998'}
# {'value': None, 'key': '9999'}

Deleting multiple documents

Sonya provides delete_many method. This method is fastest option when you want to remove multiple documents from the database. The method has cursor-like interface. The whole operation will be processed in the one transaction.

The method returns number of affected rows.

from sonya import Environment, fields, Schema


class IntSchema(Schema):
    key = fields.UInt32Field(index=0)
    value = fields.PickleField()


env = Environment('/tmp/test-env')
db = env.database('test-integer-db', IntSchema(), compression='zstd')
env.open()


with db.transaction() as tx:
    for i in range(10000):
        tx.set(db.document(key=i, value=None))

# returns the number of affected rows
db.delete_many(order='>=', key=9995):

Document count

The Database objects has a __len__ method. Please avoid to use it for any big database, it iterates and count the documents each time (faster then using len(list(db.cursor())) but still has O(n) complexity).

from sonya import Environment, fields, Schema


class IntSchema(Schema):
    key = fields.UInt32Field(index=0)
    value = fields.PickleField()


env = Environment('/tmp/test-env')
db = env.database('test-integer-db', IntSchema(), compression='zstd')
env.open()


with db.transaction() as tx:
    for i in range(10000):
        tx.set(db.document(key=i, value=None))

print(len(db))
# 10000

Transactions

Sophia supports ACID transactions. Even better, a single transaction can cover operations to multiple databases in a given environment.

Example usage:

class Users(Schema):
    name = fields.StringField(index=0)
    surname = fields.StringField(index=1)
    age = fields.UInt8Field()


with users.transaction() as tx:
    tx.set(users.document(name='Jane', surname='Doe', age=19))
    tx.set(users.document(name='John', surname='Doe', age=18))

    # Raises LookupError
    db.get(name='John', surname='Doe')

Multiple transactions are allowed to be open at the same time, but if there are conflicting changes, an exception will be thrown when attempting to commit the offending transaction.

>>> print(env['sophia.status'])
"online"

>>> env['scheduler.threads'] = env['scheduler.threads'] + 2
>>> env.open()
>>> print(env['scheduler.threads'])
8
>>> print(dict(env))
{'db.test-string-db.stat.cursor_latency': '0 0 0.0', ...}