Cython wrapper for tkvdb radix trie key-value database
Project description
python-tkvdb
Python-tkvdb is a Cython wrapper for tkvdb trie key-value database. Python 3 is required.
Code isn't well tested in production environments.
Installation
This is a typical python/cython package that uses setuptools build system.
From PyPi
The most simple way of installing is using pip
:
pip install python-tkvdb
Considering that package is using Cython, C compiler may be required for building if suitable wheel for current platform isn't found.
Downloading
Original tkvdb sources are included as submodule, so when installing
from git please use git clone --recursive
for cloning, or init
submodules after with git submodule update --init --recursive
. Custom sources may also be provided in tkvdb/
subdirectory before build.
Source archives that were made with python setup.py sdist
contain
generated C-code without .pyx
files as Cython official documentation
recommends. Cython header files (.pxd
) still provided though.
Package also can be distributed as python wheels. With wheels no additional post-installation actions are required.
Initialization and usage of virtualenv or alternatives aren't properly described in this manual, so use them by you own discretion.
Building
Project uses Cython to generate C extension for Python. To build package, you need to have C compiler and build tools.
For installation from the source archive, Cython isn't required, but
git versions require it. Source directory also includes
pyproject.toml
(PEP 518), so if your build tool uses it, Cython
would be installed anyway.
To make common tasks easier, project contains simple Makefile that may
be used instad of pip/python commands. It isn't a requirement, so pip install .
also works. For additional reference, look into
Makefile.
Install example with virtualenv:
cd python-tkvdb/
python -m venv env
source ./env/bin/activate
make
Both Makefile and setup.py uses USE_CYTHON
env variable (int, 0/1)
to determine if Cython (cythonize) would be started before extension
building. Cython needs to be installed in local environment for
this. Default value is 1 for make.
Makefile also has PYTHON
env var that allows overriding python
interpreter path. Default value is just python
.
Example usage:
USE_CYTHON=1 PYTHON=./env/bin/python make build
Make commands:
build
-- just build extension withpython setup.py build_ext
.install
-- runpip install
. Extension would be built if needed.- no arguments (just
make
) -- alias forinstall
. dist
-- create wheel and sdist archive.test
-- run unit testsclean
-- remove generated code, compiled objects and distribution archives.uninstall
-- remove previously installed package (throughpip
)
After installing module tkvdb
must be importable in the Python environment.
Usage
Original tkvdb
uses pretty specific terminology for some actions
(like transaction), so it is recommended to first consult with
original documentation anyway. Thread-safety notes and some caveats
are also described in the original README file.
Python-tkvdb provides pythonic-style wrapper for most parts of the original library.
Most object have destructors that call free
internally and do memory
management on garbage collection, but don't forget that this is
wrapper for C library.
Some code examples may also be found in tests code.
Modules
Project is splitted into multiple python modules:
ctkvdb
-- Cython wrapper with C definitions fromtkvdb.h
.tkvdb.db
-- database object and initialization. Also imported in__init__.py
(i.e. maintkvdb
module).tkvdb.transaction
-- transaction (actually main input/output. interface). Wrapper aroundtkvdb_tr
.tkvdb.cursor
-- transaction cursors for iteration. Wrapper aroundtkvdb_cursor
.tkvdb.iterators
-- pythonic iterators fortkvdb.cursor
.tkvdb.errors
-- all db-related exceptions that code may throw.tkvdb.params
-- database and transaction params. Wrapper aroundtkvdb_params
.
Database initialization
Database is wrapped into the Tkvdb
object from tkvdb
or tkvdb.db
modules. At this time only path to database file is
supported. Parameters (tkvdb.params.Params
) optionally may be passed
to constructor.
from tkvdb import Tkvdb
db = Tkvdb(path_to_db_file)
# some code
db.close()
Context manager (with
statment) that includes auto closing is also
available:
with Tkvdb(path_to_db_file) as db:
# some code
with db.transaction() as tr:
# more code
Attributes (readonly):
path: str
-- path to database file.is_opened: bool
-- shows that database is initialized properly.
Methods (may raise exceptions):
Tkvdb(path: str, params: tkvdb.params.Params = None)
(constructor) -- create database instance.close()
-- close database.transaction(params: tkvdb.params.Params = None) -> tkvdb.transaction.Transaction
-- create transaction.
There is also Cython method get_db
that returns tkvdb_db *
pointer.
Transactions
Transactions are basic way to do any operation with database. Consult with original documentation about transaction term, because it doesn't mean same thing as in other database systems.
Input and ouput uses bytes
type for everything. Encode and
decode strings if needed.
Parameters (tkvdb.params.Params
) optionally may be passed to
constructor.
Transaction must be created from database instance (described in previous part):
transaction = db.transaction()
transaction.begin()
transaction.put(b'key', b'value')
transaction.commit() # or transaction.rollback()
print(transaction.getvalue(b'key'))
transaction.free()
Pythonic with
statment also available:
with db.transaction() as tr:
tr.put(b'key', b'value')
tr.commit()
print(tr.getvalue(b'key'))
Note that with
statement does not do commit, but rollbacks on
exception. Do commit
or rollback
with your own code, or don't do
anything (implies rollback-like behavior). Transaction is started
(begin
) automatically and will be freed (free
) at exit from with
block though.
Transaction also has Python dict-like interface:
__getitem__
and__setitem__
get(key, default=None)
keys()
,values()
anditems()
iterators
with db.transaction() as tr:
tr[b'key'] = b'value'
print(tr.get(b'other-key', b'default')) # prints b'default'
tr.commit()
print(tr[b'key']) # prints b'value'
# Iterators
for key in tr: # or tr.keys()
print(key)
for key, value in tr.items():
print(key, value)
for value in tr.values():
print(value)
Multiple keys may be deleted by delete
method using optional
prefix
argument (default False
). Passing True
allows deleting
keys starting from prefix. Dict-like del
operator always deletes
only exact key.
Attributes (readonly):
is_initialized: bool
-- shows that transaction underlying structures are initialized properly.is_started: bool
-- shows thatbegin()
method was called.is_changed: bool
-- shows that transaction had any uncommited changes (i.e.put()
was used).ram_only: bool
-- indicates that transaction is RAM-only.
Transaction methods. Most of them may raise an exception:
Transaction(ram_only=True, params: tkvdb.params.Params = None)
(constructor) -- create transaction instance. Must be called manually only for RAM-only usage, otherwisedb.transaction()
must be used instead.begin()
-- starts transaction, calls underlyingtkvdb_tr->begin()
.getvalue(key: bytes) -> bytes
-- get value by key.put(key: bytes)
-- insert value into db by key.get(key: bytes, default: bytes = None) -> bytes
-- dict-like get with default value.delete(key: bytes, prefix=False)
-- delete value by key. Withprefix=True
all keys staring withkey
will be deleted.__getitem__
,__setitem__
,__delitem__
-- dict-like methods.__contains__
-- allows usage ofin
operator.free()
-- free transaction (called inwith
statement automatically).keys()
,values()
,items()
-- return dict-like iterators.cursor(seek_key=None, seek_type=Seek.EQ)
-- return transaction cursor (seeCursors
), with optional seek.
RAM-only transactions
Transactions also may be used in RAM-only mode. These transactions don't
require database file and use less memory. They are cleared on
commit()
or rollback()
. See more about RAM-only transactions in
original documentation.
Use tkvdb.Transaction()
constructor to create RAM-only transaction
without database. This transaction may also be used with with
statement, same auto-begin rules apply. Example:
with Transaction() as tr:
tr[b'key'] = b'value'
print(tr[b'key']) # prints b'value'
tr.commit() # clears transaction
Iterators
Transaction can be traversed using iterators. It is also the main way for iterating through database contents.
Module tkvdb.iterators
provides three dict-like iterators that use
tkvdb.cursor.Cursor
inside:
tkvdb.iterators.KeysIterator
-- iterating over keys.tkvdb.iterators.ValuesIterator
-- iterating over values.tkvdb.iterators.ItemsIterator
-- iterating over key-value pair.
They can be used with transaction:
with db.transaction() as tr:
for key in tr: # or tr.keys()
print(tr[key])
for value in tr.values():
print(value)
for key, value in tr.items():
print(key, value)
In all loops new instance of Cursor
is used.
They also can be used with cursor:
with db.transaction() as tr:
with tr.cursor() as c:
for key in c:
print(c.key(), c.keysize())
Notice that cursor iterators use same underlying Cursor
object, so
they would iterate from same place where cursor stopped before:
with db.transaction() as tr:
with tr.cursor() as c:
c.first()
# do some iteration with c.next()
for key in c:
print(key) # it wouldn't be first key
if something:
break
for value in c.values(): # starts from last iterated key
print(value)
Reverse iteration is available through the standard reversed
function. Iterators and transaction have required methods for this.
For new cursors iteration will start from the end using
tkvdb.cursor.Cursor.last
.
with db.transaction() as tr:
for key in reversed(tr):
print(key)
# All iterator types allow this
for key, value in reversed(tr.items()):
print(key, value)
with db.transaction() as tr:
with tr.cursor() as c:
for key in reversed(c):
print(key)
with tr.cursor() as c:
for value in reversed(c.values()):
print(value)
Cursors created from transacton can used for searching (as shorthand
for tkvdb.cursor.Cursor.seek()
). More information in next section.
with db.transaction() as tr:
with tr.cursor(seek_key=b'seek-tr-31', seek_type=Seek.GE) as c:
print(c.key())
Cursors
Cursors are used to iterate through database contents. They are
defined in tkvdb.cursor
module, C implementation is wrapped in
tkvdb.cursor.Cursor
class.
Cursors are attached to transaction and created by
Transacion.cursor()
method. They also may be created directly.
Although cursors are sole way to iterate and seek in tkvdb, it is better and easier to use python-style iterators for such purposes.
Example usage:
with db.transaction() as tr:
with tr.cursor() as c:
c.first()
while True:
print(c.key(), c.value())
try:
c.next()
except tkvdb.errors.NotFoundError:
break
Cursor also may be used without with
statement, it would be freed
anyway on garbage collection:
with db.transaction() as tr:
c = tr.cursor():
c.first()
# ...
Notice: first
and next
methods throw tkvdb.errors.EmptyError
on
empty database, not NotFoundError
. Cursors may be iterated by using
iterators (see previous section).
Cursor also may be iterated in reverse order using prev()
method. Another method called last()
moves cursor to last record and
often useful for the reverse iteration.
Cursor can be used for search with seek()
method. This allows
searching k-v pair by prefix using seek criteria. Criterias are
defined in tkvdb.cursor.Seek
enum:
Seek.EQ
-- search for the exact key match.Seek.LE
-- search for less (in terms of memcmp()) or equal key.Seek.GE
-- search for greater (in terms of memcmp()) or equal key.
Seeking is also may be initiated using Transaction.cursor()
method
(see more in Transactions
section).
from tkvdb.cursor import Seek
with db.transaction() as tr:
with tr.cursor() as c:
c.seek(b'key', Seek.EQ)
key = c.key()
# ...
c.next()
with tr.cursor(seek_key=b'key') as c:
# ...
Attributes (readonly):
is_initialized: bool
-- shows that cursor underlying structures are initialized properly.is_started: bool
-- shows thatfirst()
method was called.
Cursor methods.
first()
-- move cursor to first item in database.last()
-- move cursor to last item in database.next()
-- move cursor to next item in database.prev()
-- move cursor to previous item in database.key() -> bytes
-- get current key.val() -> bytes
-- get current value.keysize() -> int
-- get current key size.valsize() -> int
-- get current value size.free()
-- free cursor.__iter__()
-- returnstkvdb.iterators.KeysIterator
.seek(key: bytes, seek: tkvdb.cursor.Seek)
-- search key by criteria.keys()
,values()
,items()
-- return dict-like iterators.
Params
Params are used to specify different options for database and/or
transactions. They are defined in tkvdb.params
module: the C
implementation (tkvdb_params
struct) is wrapped by
tkvdb.params.Params
class, and param values are wrapped by the
tkvdb.params.Param
enum.
Parameter names transfomed using CamelCase, example:
TKVDB_PARAM_TR_DYNALLOC => TrDynalloc
TKVDB_PARAM_CURSOR_STACK_DYNALLOC => CursorStackDynalloc
Available parameters in tkvdb.params.Param
enum:
TrDynalloc
--TKVDB_PARAM_TR_DYNALLOC
TrLimit
--TKVDB_PARAM_TR_LIMIT
Alignval
--TKVDB_PARAM_ALIGNVAL
Autobegin
--TKVDB_PARAM_AUTOBEGIN
CursorStackDynalloc
--TKVDB_PARAM_CURSOR_STACK_DYNALLOC
CursorStackLimit
--TKVDB_PARAM_CURSOR_STACK_LIMIT
CursorKeyDynalloc
--TKVDB_PARAM_CURSOR_KEY_DYNALLOC
CursorKeyLimit
--TKVDB_PARAM_CURSOR_KEY_LIMIT
DbfileOpenFlags
--TKVDB_PARAM_DBFILE_OPEN_FLAGS
Consult with original tkvdb
documentation for params meaning and
possible values.
Params usage:
from tkvdb.params import Params, Param
# Passing params to database
params = Params({Param.Autobegin: 1}) # set params at init
with Tkvdb(path, params) as db:
# Params also will be passed to db transactions
with db.transaction() as tr:
# ...
# Transaction may use own params
params = Params()
params.set(Param.TrLimit, 100)
with db.transaction(params) as tr:
# ...
# Setting params after init
params = Params()
params.set(Param.Autobegin, 1)
tr = Transaction(params=params) # RAM-only transaction
As in original tkvdb
code, params from database are passed to
db-bound transaction, if they aren't overrided directly by passing
another Params
instance to transaction.
Python implementation also stores all set params and stores them in
internal values
dict. Notice that it tracks only params that were
set directly, so default values aren't known to Params
wrapper.
Params attributes:
is_initialized: bool
-- shows that params underlying structures are initialized properly.
Params methods:
Params(params=None)
(constructor) -- create params instance. Argumentparams
may be dict with param names and values.get_values()
-- return all set params.get(param: tkvdb.params.Param)
-- return single param value.set(param: tkvdb.params.Param, value: int)
-- set param value.free()
-- free params object.
There is also Cython method get_params
that returns tkvdb_params *
pointer.
Errors
Error classes are defined in tkvdb.errors
module. Every non-ok
return value from the underlying C code is converted to python
Exception. Only TKVDB_RES.TKVDB_OK
considered as success.
Consult with original documentation for error codes meaning.
One exception from this rule is
tkvdb.transaction.Transaction.__getitem__()
(dict-like access) that
raises KeyError
for python compatibility.
Examples:
from tkvdb.errors import EmptyError, NotFoundError, NotStartedError
# ...
tr = db.transaction()
try:
print(tr.getvalue(b'key'))
except (NotFoundError, EmptyError):
print('key not found')
except NotStartedError:
print('transaction not started')
with db.transaction() as tr:
try:
print(tr[b'key'])
except KeyError:
print('key not found')
Note that tkvdb raises EmptyError
(TKVDB_RES.TKVDB_EMPTY
return
code), not NotFoundError
when key is not found in empty database,
Errors:
Error
-- base class for all tkvdb-related errors.IoError
--TKVDB_RES.TKVDB_IO_ERROR
code.LockedError
--TKVDB_RES.TKVDB_LOCKED
code.EmptyError
--TKVDB_RES.TKVDB_EMPTY
code.NotFoundError
--TKVDB_RES.TKVDB_NOT_FOUND
code.EnomemError
--TKVDB_RES.TKVDB_ENOMEM
code.CorruptedError
--TKVDB_RES.TKVDB_CORRUPTED
code.NotStartedError
--TKVDB_RES.TKVDB_NOT_STARTED
code.ModifiedError
--TKVDB_RES.TKVDB_MODIFIED
code.
License
Python-tkvdb is licensed under ISC license as original tkvdb project.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for python_tkvdb-0.2.3-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1313435f6550564a0c6a01ee4d4cb9bd3ace561cc73e8243efaaa40bb7cd46f5 |
|
MD5 | 237bd78ccbf942673f20f84410f69e4a |
|
BLAKE2b-256 | 90295330e78cbaba97158358a5884848446493f4ec9935e288bd3b99fa77acb6 |
Hashes for python_tkvdb-0.2.3-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 689f2a68d800998c66aa873f0bf00d101e611a31f8609f3e3bcc37a789381908 |
|
MD5 | e52a8272f5295d902044cc54516351a0 |
|
BLAKE2b-256 | a09aeafd29121e2a19b575987deb0f2786ccd5572e1325b9eb0e6d800323fe87 |
Hashes for python_tkvdb-0.2.3-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 19e14d60a5f206935181da7f31e9f9616ab1aa90aac2f1a173a2380f256754d1 |
|
MD5 | 78dacb3bd3c84cfdc80ea2a7201535e1 |
|
BLAKE2b-256 | 3f00b26b6181163e90eda5a7bbfcd45413762b49b30fe1621d6dc911374e4ccc |
Hashes for python_tkvdb-0.2.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8a8e44ae35e6187b73d9257a56146ff8722eade5aa28146989dd511997cb56c8 |
|
MD5 | 33026ecadd1e0ce14a4a324019ed11d6 |
|
BLAKE2b-256 | 64fc1344e0894dce63800f5eb8c68317697f346969ec35b831fc289b7ab483d5 |
Hashes for python_tkvdb-0.2.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bf45449f53ec32493b5532fdb45a60059897a4c3035d2bc634c7a122a344ae1d |
|
MD5 | f22da44ad2e19df938f4f3ad16431420 |
|
BLAKE2b-256 | a6c445e32be4c86158b8c7f6f755af6924b5bff9c2e2178d035a4d95cd33d092 |
Hashes for python_tkvdb-0.2.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 03dee3e00c2c7821b1f545b26101a4f3066e2765f82a3d248d86a4de533f30ec |
|
MD5 | 4113a72d3421478a8e6832ee0f2531e6 |
|
BLAKE2b-256 | cab9325ca1d401f22d65d5cba4f351fc57a83c9aa382c3e5f894ad7f2ff57834 |
Hashes for python_tkvdb-0.2.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fd464f2da0c3d44c3364a677da216441306cdc55059a9a3b2779023d64f40631 |
|
MD5 | c551bbd8806045d6965aaac4fffadfd5 |
|
BLAKE2b-256 | 4c92cdaadf4a200395071b1f2ebb5c1893e85c99f222e90fbde3c00180e7de69 |
Hashes for python_tkvdb-0.2.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7e953116d6aa2c6b7e304929b81983349949f572ac582d2422fafb0944e5eb1e |
|
MD5 | f18df27ea7ce423e79546d43e679c0e2 |
|
BLAKE2b-256 | 4a83d06cc4f6139712afaeae0bb361f713ad1da1fd8f9c9237aef1cb3e289016 |
Hashes for python_tkvdb-0.2.3-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a2754ce78eb5c0c6e22191ba47203019fcb9880c97984d0475bcd60cb683fa90 |
|
MD5 | a07d997f00d3596d9e645fcbe3f31bf9 |
|
BLAKE2b-256 | 218c59447ab390917378920dbdf222a486c781eb8826c7d6ab0c864f0481c655 |