pyrex

Installation

pyrex-rocksdb

A python wrapper for the original (C++) version of RocksDB.

Currently MacOS and Linux wheels are available.

Installation

For linux systems, wheels are provided and can be installed from pypi using:

pip install pyrex-rocksdb

For Windows and MacOS I have built an earlier version of the library. I will re-build once I include certain other important features in the API that are not yet implemented.

Motivation

This library is intended for providing a fast, write-optimized, in-process key value (KV) store in python. Therefore the "big brothers" of the database are the likes of MongoDB and Cassandra. The difference is that you don't need a separate server to run this (hence "in-process") and it is designed to be fairly portable.

RocksDB, which is the underlying storage engine of this database, is an LSM-tree engine. An LSM-tree is different from the ballanced tree index databases (e.g., B-tree/ and B+tree databases). LSM-tree databases offer very high write throughputs and better space efficiency. See more about the motivation for LSM-tree databases (and RocksDB in particular) in this talk.

LSM-tree + SSTable engine basics

To understand where pyrex provides efficiency gains, it is important to understand some basics about the underlying RocksDB engine.

RocksDB and LevelDB are key-value stores with a Log-Structured Merge-tree (LSM-tree) architecture.

The key components of LSM-tree architectures are

A MemTable that stores in-memory sorted data
A set of Sorted-String tables (SSTables) which are immutable sorted files on disk where data from the MemTable is flushed
The process of Compaction, which is a background process that merges the SSTables to remove redundant data and keep read performance high.

In such databases, fast writes create many small, sorted data files called SSTables. To prevent reads from slowing down by checking too many files, a background process called compaction merges these SSTables together. This process organizes the data into levels, where newer, overlapping files sit in Level 0 and are progressively merged into higher levels (Level 1, Level 2, etc.). Each higher level contains larger, non-overlapping files, which ensures that finding a key remains efficient and old data is purged to save space. There are several optimizations and configurations possible for these processes (configurability and "pluggability" are commonly cited RocksDB advantages).

However the main big advantage of RocksDB over LevelDB is its multi-threaded compaction support (LevelDB supports only single threaded compaction, which comes with significant performance limitations). There are several other configurability advantages RocksDB offers over LevelDB. For a more elaborate enumaration of RocksDB advantages please refer to the RocksDB wiki.

Not all are currently supported by the pyrex API, but I'm working on supporting more of them. Feel free to open an issue if there is a feature you want to see (or open a pull request).

Example usage:

Here is a simple example showing the usage of put/get in the DB:

import pyrex
import os
import shutil

DB_PATH = "./test_rocksdb_minimal"

with pyrex.PyRocksDB(DB_PATH) as db:
    db.put(b"my_key", b"my_value")
    retrieved_value = db.get(b"my_key")

print(f"Retrieved: {retrieved_value.decode()}") # Output: Retrieved: my_value

for more examples check the relevant folder and the documentation.

Native columnar batch writes

For Arrow-compatible binary or string arrays, write_columnar_batch moves the per-row batch construction loop into C++:

import pyarrow as pa
import pyrex

keys = pa.array([b"k1", b"k2", b"k3"], type=pa.binary())
values = pa.array([b"v1", b"v2", b"v3"], type=pa.binary())

with pyrex.PyRocksDB("example_columnar_db") as db:
    db.write_columnar_batch(keys, values)

Polars users can pass series.to_arrow(); Polars is not a required dependency.

Transactions

Use TransactionDB when a group of reads and writes must commit or roll back together. Transaction context managers use explicit commit semantics: if commit() is not called, the transaction rolls back when the block exits.

import pyrex

with pyrex.TransactionDB("example_txn_db") as db:
    with db.transaction() as txn:
        txn.put(b"k", b"v")
        assert txn.get(b"k") == b"v"
        txn.commit()

    assert db.get(b"k") == b"v"

Existing PyWriteBatch objects can be applied inside a transaction:

batch = pyrex.PyWriteBatch()
batch.put(b"a", b"1")
batch.delete(b"old")

with pyrex.TransactionDB("example_txn_db") as db:
    with db.transaction() as txn:
        txn.write(batch)
        txn.commit()

Transaction iterators can be used for prefix/range scans and include transaction-local writes where RocksDB supports them:

with pyrex.TransactionDB("example_txn_db") as db:
    with db.transaction() as txn:
        txn.put(b"user:1", b"alice")
        it = txn.new_iterator()
        it.seek(b"user:")
        while it.valid() and it.key().startswith(b"user:"):
            print(it.key(), it.value())
            it.next()
        txn.rollback()

WriteOptions.disable_wal is preserved for transaction writes and commits, but disable_wal=True is not fully durable across crashes. Transaction conflicts and lock timeouts are exposed through specific exception subclasses such as RocksDBBusyError, RocksDBTimeoutError, and RocksDBConflictError.

Note on CICD

The windows wheels are failing at the moment. The CICD workflow for package builds works and passes all tests only for MacOS and Linux.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

pyrex_rocksdb-0.4.0-cp312-cp312-musllinux_1_2_x86_64.whl (16.3 MB view details)

Uploaded Aug 16, 2026 CPython 3.12musllinux: musl 1.2+ x86-64

pyrex_rocksdb-0.4.0-cp312-cp312-musllinux_1_2_aarch64.whl (6.8 MB view details)

Uploaded Aug 16, 2026 CPython 3.12musllinux: musl 1.2+ ARM64

pyrex_rocksdb-0.4.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (15.1 MB view details)

Uploaded Aug 16, 2026 CPython 3.12manylinux: glibc 2.27+ x86-64manylinux: glibc 2.28+ x86-64

pyrex_rocksdb-0.4.0-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl (4.5 MB view details)

Uploaded Aug 16, 2026 CPython 3.12manylinux: glibc 2.26+ ARM64manylinux: glibc 2.28+ ARM64

pyrex_rocksdb-0.4.0-cp312-cp312-macosx_14_0_arm64.whl (3.7 MB view details)

Uploaded Aug 16, 2026 CPython 3.12macOS 14.0+ ARM64

pyrex_rocksdb-0.4.0-cp311-cp311-musllinux_1_2_x86_64.whl (16.2 MB view details)

Uploaded Aug 16, 2026 CPython 3.11musllinux: musl 1.2+ x86-64

pyrex_rocksdb-0.4.0-cp311-cp311-musllinux_1_2_aarch64.whl (6.8 MB view details)

Uploaded Aug 16, 2026 CPython 3.11musllinux: musl 1.2+ ARM64

pyrex_rocksdb-0.4.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (14.9 MB view details)

Uploaded Aug 16, 2026 CPython 3.11manylinux: glibc 2.27+ x86-64manylinux: glibc 2.28+ x86-64

pyrex_rocksdb-0.4.0-cp311-cp311-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl (4.5 MB view details)

Uploaded Aug 16, 2026 CPython 3.11manylinux: glibc 2.26+ ARM64manylinux: glibc 2.28+ ARM64

pyrex_rocksdb-0.4.0-cp311-cp311-macosx_14_0_arm64.whl (3.7 MB view details)

Uploaded Aug 16, 2026 CPython 3.11macOS 14.0+ ARM64

pyrex_rocksdb-0.4.0-cp310-cp310-musllinux_1_2_x86_64.whl (16.1 MB view details)

Uploaded Aug 16, 2026 CPython 3.10musllinux: musl 1.2+ x86-64

pyrex_rocksdb-0.4.0-cp310-cp310-musllinux_1_2_aarch64.whl (6.8 MB view details)

Uploaded Aug 16, 2026 CPython 3.10musllinux: musl 1.2+ ARM64

pyrex_rocksdb-0.4.0-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (14.9 MB view details)

Uploaded Aug 16, 2026 CPython 3.10manylinux: glibc 2.27+ x86-64manylinux: glibc 2.28+ x86-64

pyrex_rocksdb-0.4.0-cp310-cp310-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl (4.5 MB view details)

Uploaded Aug 16, 2026 CPython 3.10manylinux: glibc 2.26+ ARM64manylinux: glibc 2.28+ ARM64

pyrex_rocksdb-0.4.0-cp310-cp310-macosx_14_0_arm64.whl (3.7 MB view details)

Uploaded Aug 16, 2026 CPython 3.10macOS 14.0+ ARM64

pyrex_rocksdb-0.4.0-cp39-cp39-musllinux_1_2_x86_64.whl (16.1 MB view details)

Uploaded Aug 16, 2026 CPython 3.9musllinux: musl 1.2+ x86-64

pyrex_rocksdb-0.4.0-cp39-cp39-musllinux_1_2_aarch64.whl (6.8 MB view details)

Uploaded Aug 16, 2026 CPython 3.9musllinux: musl 1.2+ ARM64

pyrex_rocksdb-0.4.0-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (14.8 MB view details)

Uploaded Aug 16, 2026 CPython 3.9manylinux: glibc 2.27+ x86-64manylinux: glibc 2.28+ x86-64

pyrex_rocksdb-0.4.0-cp39-cp39-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl (4.5 MB view details)

Uploaded Aug 16, 2026 CPython 3.9manylinux: glibc 2.26+ ARM64manylinux: glibc 2.28+ ARM64

pyrex_rocksdb-0.4.0-cp39-cp39-macosx_14_0_arm64.whl (3.7 MB view details)

Uploaded Aug 16, 2026 CPython 3.9macOS 14.0+ ARM64

File details

Details for the file pyrex_rocksdb-0.4.0-cp312-cp312-musllinux_1_2_x86_64.whl.

File metadata

Download URL: pyrex_rocksdb-0.4.0-cp312-cp312-musllinux_1_2_x86_64.whl
Upload date: Aug 16, 2026
Size: 16.3 MB
Tags: CPython 3.12, musllinux: musl 1.2+ x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.14.7

File hashes

Hashes for pyrex_rocksdb-0.4.0-cp312-cp312-musllinux_1_2_x86_64.whl
Algorithm	Hash digest
SHA256	`11fd46256175e2f491ca01fa137184474ceb34379cca347131ca1a59c8a17695`
MD5	`a091205b9591d173de5c5215e19f861e`
BLAKE2b-256	`fcb1dd76dc6f3fc101c10e7c809b1a841c7c973a0c2d27992cfa4766c44911b4`

Algorithm	Hash digest
SHA256	`0a53e5589d50d80cf8647ea364bc6fad432de1cd19cb9d9d80e7fb1ab6f97669`
MD5	`ec029b70576a30f9cd76698431265004`
BLAKE2b-256	`eb8b1b9b4a73842e6e5aec9c547c87f518acd452052624f7c05259a703adfda0`

Algorithm	Hash digest
SHA256	`d63b688d51344cd363dfcdde986facc1dc891b20fee36afabe40a47b624d72ce`
MD5	`e7331b472e115396428695cd6cf71512`
BLAKE2b-256	`b88469cf63348160778eb9bc9ac9c77c5a05d066d027b1f67a5121a8fa39f3fe`

Algorithm	Hash digest
SHA256	`b3f72f70ff6aa8afe2ae2ede13e43e5fba7001df9af48ceaa111e18c5072ea62`
MD5	`7f369504da4b79f00a2f9d52e212913a`
BLAKE2b-256	`d43530b8cbea34c1da10a448a1108d82c3914971c31f0d1a5a40335c1c694e58`

Algorithm	Hash digest
SHA256	`2a171b73ab89bcb57927bed0f4e656536abf84a9dc861aa227981f2862aec7f6`
MD5	`a1130ffa2c06b9fd553c4a84df94b28d`
BLAKE2b-256	`85253153b37b6f52710e8809eec90f1c002481ee1d8b04edcbc8d3b7a2f6fc66`

Algorithm	Hash digest
SHA256	`650a210738be72e89f20d1bd555b939c7f197b216a309d725f75eb44b6dbb068`
MD5	`73cbefb6eafbd2dd4d1588286596e383`
BLAKE2b-256	`7c4ebad4b84046536a7ec684a5d5d51a8d0bbc0c1b7c120690f3bc17311c4dea`

Algorithm	Hash digest
SHA256	`9eeab6f3066fe09b3339bd644d92ef1a79c097173d338a724b3db4d8b732f321`
MD5	`0109ce44636f8abd5eeb2c115a80f0c2`
BLAKE2b-256	`3d059dac1a5b9802cbc8809113d1a2fcce854387759865a9bf19d56d36d207f5`

Algorithm	Hash digest
SHA256	`5fe084c64bb03ab76aa3fdbc1f8af9b4b5f452acfbdaeeb0fd730ee1f39ac0b3`
MD5	`68bd4a61960a29138210e0ad16cafb10`
BLAKE2b-256	`91017e113d50f7e976c8f0ea92988b7d6358bac8ae03441fd38c447c40b761e9`

Algorithm	Hash digest
SHA256	`8ca28c50501af1a5e63da37f04501ad3126b775ea74343f6f4072b1ed41c9c02`
MD5	`6eb9eab47044e925019d9bb5fba340a6`
BLAKE2b-256	`4067a3c842bbc971a46810cec5c6a5323f7360ff27202393c0fb7b447b7e08bf`

Algorithm	Hash digest
SHA256	`e2c0e75fcb58a78f47518acf9487a1fe236032798fcca28015d7e82d6e352818`
MD5	`cd47803d5314b7d858bccd88808a6f2c`
BLAKE2b-256	`f8d3c664198dade1c977daba82ce86d70795e5f0c81a41585b7f66dc3d64f775`

Algorithm	Hash digest
SHA256	`2b696a3c43a1a091cc28051926ebb0031357e229140eb362bbf0f15cc7d45778`
MD5	`6cf15bf4de00146b53bbbb4660f04a80`
BLAKE2b-256	`c8a7ea7a8a94ec612017d00bb00c58a23785dff8244195d42fce22a2e093eb60`

Algorithm	Hash digest
SHA256	`53407a59813497630b2010a4faea14b9d2005457533a7ed48b73a805c2c663bd`
MD5	`5627471f150a539041228a3ec203b7d1`
BLAKE2b-256	`1ce13f7ef1b6ac749ce34f0713315a29ef1cffe3b9fc4debc2676e98a7695369`

Algorithm	Hash digest
SHA256	`0c33be8900cd99d39d0e0b62970cf8a06733933bf06f01a6c6f1a3d5bc09c61c`
MD5	`3893ddd038da9a403456db1436deb2da`
BLAKE2b-256	`24eaebb281132dcfa6f137b4b72dae9c36f724ea6071eaa234fefb1dcecb7f1d`

Algorithm	Hash digest
SHA256	`bcc86299e0f1ac407ae9b752e2e3a5be2743212e074441dd1b846287bac9f916`
MD5	`b56ed93950af47d2c3aaed8b1257d086`
BLAKE2b-256	`f4d6613e73d008f867cebb276967be0a8b4d2dbab9e997970283840df916afa9`

Algorithm	Hash digest
SHA256	`7d46b3f0466b86df2479b7035bb606c60b7ef4184b5be2ec44c1ba595b4d9181`
MD5	`a6bfe2212a2a86a956796930d8401be3`
BLAKE2b-256	`af1cf4a13ab08ccad89c5333a4999ea2b105110a60f388e0043df74bb69121c0`

Algorithm	Hash digest
SHA256	`2a85029650b5b4ec6f1285070488458fd7c10b0b3ab0686673f7f704f803eae1`
MD5	`89edc085c3bd4ef6944fda9e67496998`
BLAKE2b-256	`9072c2222c8df8d3d8a158260ef638ad4c65fec5f5ee56c1b8036dc30300827b`

Algorithm	Hash digest
SHA256	`c19928b60a7531a8bb43f1208cddf296b7956684984c542b4f9169d28f933542`
MD5	`24e2b261c30256d6ad1263625e74d6d4`
BLAKE2b-256	`3464574f15343364f36d64f768374909f0541d11a44faf97e2333331fe83df9a`

Algorithm	Hash digest
SHA256	`22692c23ebf377b81637444fb2489611d0753f65cc3a087aa31b705ffdba10e3`
MD5	`69c7dceb8f518aec9369a0f0587faffe`
BLAKE2b-256	`3cd6492411bc521e4cb146e884f23e05ba16eb481acb98357fc2f1465902023e`

Algorithm	Hash digest
SHA256	`fad8fab9ae5fa869e17bab65cfb167a277d1bf35f76dd483335797665db3fb96`
MD5	`9d1bd550ac97a2df008b4bff14dd809a`
BLAKE2b-256	`5096f28c4cc49f6fd6f22d6638c7c1847cc558775969c3d69754550b2294aa42`

Algorithm	Hash digest
SHA256	`1c7d06356caf7a950e36d55ff7e856433d9bb5881de3c132caaff0ed0138e86e`
MD5	`60c3550534adc77ade78618e81c32ff5`
BLAKE2b-256	`6bd2185064041a31feace39188091906945a6ffb6f18362d5b83f6ccdd5d5723`

pyrex-rocksdb 0.4.0

Installation

pyrex-rocksdb

Installation

Motivation

LSM-tree + SSTable engine basics

Example usage:

Native columnar batch writes

Transactions

Project links

Key dates

1 maintainer

Credits

GitHub Statistics

License

Requires

Provides Extra

Tags

Classifiers

Download files

Source Distributions

Built Distributions

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details