kydb — the Kinyu Database

One intuitive, Pythonic interface. Many NoSQL backends. Change your database by changing a URL.

kydb is an abstraction layer over NoSQL databases, built for the kind of workloads found in financial services: object hierarchies, layered caches, market-data invalidation, and objects that must survive years of refactoring.

import kydb

db = kydb.connect('dynamodb://my-table')
db['/trading/books/MYBOOK'] = my_book_object

That is the whole API surface you need to learn. Everything below is the same db object, pointed at a different URL.

Installation

pip install kydb

Why kydb

Databases in one line each

db = kydb.connect('s3://my-kydb-bucket')             # AWS S3
db = kydb.connect('dynamodb://kydb')                 # AWS DynamoDB
db = kydb.connect('redis://cache.epythoncloud.io')   # Redis (port optional)
db = kydb.connect('memory://cache001')               # In-memory
db = kydb.connect('files://tmp/foo/bar')             # Local filesystem
db = kydb.connect('http://my-source-host')           # HTTP  (read-only source)
db = kydb.connect('https://my-source-host')          # HTTPS (read-only source)

Every one of them returns the same KYDBInterface. Your code does not know, and does not care, which one it got.

def save_report(db, name, report):
    db[f'/reports/{name}'] = report   # works against all of the above

Prefer paths relative to a root? Set a base path in the URL:

db = kydb.connect('dynamodb://my-source-db/home/tony.yum')
db['foo'] = 'Hello World!'   # writes to /home/tony.yum/foo

Reading and writing

db['/mytest/foo'] = 123
db['/mytest/foo']            # 123 — served from cache, no round trip
db.read('/mytest/foo', reload=True)   # force a read from the database
db.exists('/mytest/foo')     # True
db.delete('/mytest/foo')

Any pickleable object goes in as-is:

from datetime import datetime

db['/mytest/bar'] = {
    'my_int': 123,
    'my_float': 123.456,
    'my_str': 'hello',
    'my_list': [1, 2, 3],
    'my_datetime': datetime.now(),
}

A filesystem on a document store

Document databases have no folders. kydb gives you them anyway — including empty ones — on every backend.

db.mkdir('/empty-folder')
db.ls('/')                    # ['empty-folder/']

db['/foo/bar/baz'] = 123      # intermediate folders are created for you
db.ls('/')                    # ['empty-folder/', 'foo/']
db.ls('/foo')                 # ['bar/']

db.rmdir('/empty-folder')     # removes an empty folder
db.rm_tree('/foo')            # recursive, like rm -rf
db.ls('/')                    # []

Folders always end in /; objects never do. list_dir is the lazy, paginated generator behind ls.

Unions: layer databases with a URL

Separate the databases with ;. Reads try each database in order and return the first hit; writes always go to the front.

db = kydb.connect('memory://scratch;s3://team-shared;s3://firm-golden')

db['/curves/USD.OIS']      # found in whichever layer has it first
db['/curves/USD.OIS'] = c  # always written to memory://scratch

That is a personal overlay on top of shared data, with no branching logic in your code — perfect for "what if I tweak this one curve?" research against a production dataset you must not touch.

CacheDB: a fast cache in front of a slow store

Separate two databases with |. Reads hit the cache first and fall back to the persistent store; writes go to both.

db = kydb.connect('redis://my-redis-host|s3://my-s3-prod-source')

db['/refdata/instruments'] = instruments   # written to Redis *and* S3
db['/refdata/instruments']                 # served from Redis

Cache context: invalidate exactly what you mean to

kydb caches reads in memory, so repeatedly pricing a book does not hammer the database. But live market data must not be cached across ticks. refresh() is too blunt — it drops the positions too — and tracking invalidation by hand is error-prone. Scope the cache to a with block instead:

db = kydb.connect('dynamodb://tradingdb')
book = db['/Books/MyBook']

with db.cache_context():

    positions = book.Positions()      # loaded once, cached for the outer block

    while keep_pricing():

        with db.cache_context():
            for qty, inst in positions.items():
                print(qty, inst, inst.Price())

        wait_for_next_marketdata_tick()
        # inner block exited: all market-data cache is gone

# outer block exited: the position cache is gone too

Recency queries

List a folder in modification-time order, newest first. DynamoDB uses a folder-time-index GSI and Redis a native sorted set — the ordering is done by the database, not in your process.

newest = db.recent('/articles', limit=10)     # lazy generator of names

For more control, build a query. Each call returns a new query, so a partly built one can be reused and branched:

base = db.folder('/articles').by('mtime')

names    = base.desc().limit(10)             # ['2026-09-02-launch', ...]
pairs    = base.since(timestamp_ns).items()  # (name, value) pairs
entries  = base.desc().entries()             # Entry: .key, .mtime, .ctime

Timestamps are integer nanoseconds since the Unix epoch. Backends with no server-side ordering index (Memory, Files, S3) refuse to silently run an expensive scan — you have to ask for it:

db.recent('/articles', limit=10)                    # raises IndexNotSupported
db.recent('/articles', limit=10, allow_scan=True)   # client-side sort, opt-in

Objects written before the index existed are reported honestly at mtime == 0 rather than being guessed at, and move into place the first time they are rewritten. No migration is required. See the recency documentation for the full ordering contract and the DynamoDB schema.

Business keys: order by what an object means

mtime answers when was this written. A gym takes a booking on Tuesday for Saturday's 18:30 HIIT class — and the roster the front desk needs every morning is ordered by the class date, not by when the phone rang.

Record a business key on the write, and query it with the same builder:

db.set('/signups/anna', booking, index={'class_date': 20260905})

# who is booked into Saturday's classes — one indexed query
db.folder('/signups').by('class_date').since(20260905).until(20260905).items()

# rebooking Anna to Monday: one write, and her key never moves
db.set('/signups/anna', booking, index={'class_date': 20260907})

# and mtime still answers its own question, unchanged
db.recent('/signups', limit=10)          # who booked most recently

Values are int (a date is int(d.strftime('%Y%m%d'))), both bounds are inclusive, and a rewrite that does not mention an index preserves it — index={'class_date': None} is how you clear one. The index is strictly sparse: an object with no class_date is not a badly-dated booking, it is not a booking, so it appears in no roster. DynamoDB serves each business key from its own folder-<name>-index GSI; Redis and Memory need no schema at all.

Refactor-proof objects with DbObj

Pickling a class ties your stored data to the module path it was written from. DbObj breaks that link: the database holds a name, and a config maps that name to wherever the class lives today. Move the class, and old objects still deserialise.

Register the classes once:

db = kydb.connect('memory://decorated_py_obj')

db.upload_objdb_config({
    'Greeter': {
        'module_path': 'path.to.module',
        'class_name': 'Greeter',
    }
})

Declare which attributes are persisted with @kydb.stored — everything else is transient state, rebuilt by init() on each load:

import kydb

class Greeter(kydb.DbObj):

    def init(self):
        self.greet_count = 0      # transient

    @kydb.stored
    def name(self):
        return 'John'             # the default value

    def greet(self):
        self.greet_count += 1
        return 'Hello ' + self.name()

key = '/hello-world/greeter001'

greeter = db.new('Greeter', key)
greeter.name()                     # 'John'  — the default
greeter.greet()                    # 'Hello John'

greeter = db.new('Greeter', key, name='Tony')
greeter.greet()                    # 'Hello Tony'

greeter.name.setvalue('Jane')
greeter.greet()                    # 'Hello Jane'

greeter.get_stored_dict()          # {'name': 'Jane'} — only `name` persists

Persist it and read it back; greet_count resets because it was never stored:

db[key] = greeter

db.read(key, reload=True).greet_count   # 0
db.read(key, reload=True).name()        # 'Jane'

Secured Redis

Redis connection details, including a password encrypted with AWS KMS, can be supplied through a config file pointed at by KYDB_CONFIG_PATH:

dbs:
  my-redis-db:
    host: cache.epythoncloud.io
    port: 6379
    password:
      encryption-method: kms          # or `plain`
      encryption-key: <kms-key-id>
      env_var: MY_REDIS_PASSWORD      # holds the base64 ciphertext

The same file switches off the recency index per database with mtime-index: false.

Documentation

Full documentation, including the complete API reference and the DynamoDB table schema, lives at kydb.readthedocs.io.

License

MIT — see LICENSE.