lxcore 0.2.0

Elite lock-free binary key-value database: append-only segments, snapshots, background compaction, LRU cache, tunable durability

⬡ lxcore

Elite embedded key-value database for Python.
Append-only, lock-free reads, crash-safe, segment-based, with snapshots, background compaction, and tunable durability — all in pure Python, zero dependencies.

---

At a glance

from lxcore import LxDB

with LxDB("mydb") as db:
    db.set("user:1", {"name": "Alice", "score": 42})
    db.set("user:2", {"name": "Bob",   "score": 91})

    print(db.get("user:1"))           # {'name': 'Alice', 'score': 42}
    print(db.prefix_scan("user:"))    # [(b'user:1', {...}), (b'user:2', {...})]

---

Design

lxcore is built around one principle: deterministic performance with predictable I/O.

Property	Detail
Storage	Append-only segment files (seg_00000001.lx, …)
Index	In-memory hash map + persistent checkpoint
Concurrency	Single writer, unlimited lock-free readers
Durability	Three modes: unsafe, normal, strict
Compaction	Background daemon + manual trigger
Compression	Optional per-value zlib (level 1)
CRC	CRC-32 on every record — detects corruption on read
Snapshots	Immutable read views captured at a point in time

No locks, no SQL, no query planner, no external dependencies.

---

Installation

pip install lxcore

Requires Python ≥ 3.10.

---

Quick start

Open / close

from lxcore import LxDB

db = LxDB("mydb")        # creates mydb.lxdb/ if absent
db.close()

# context manager (preferred)
with LxDB("mydb") as db:
    ...

Write

db.set("key", "value")           # typed upsert (any JSON-compatible value)
db.set("key", 42)
db.set("key", {"a": [1, 2, 3]})
db.set("key", b"\xde\xad")       # raw bytes

db.create("newkey", "value")     # raises KeyError if key exists
db.write("key", b"rawbytes")     # raw upsert

Read

val = db.get("key")              # typed read — returns Python object
raw = db.read("key")             # raw bytes
ok  = db.exists("key")           # bool

Update / Delete

db.update("key", "new_value")    # raises KeyError if missing
db.delete("key")                 # returns True/False

Batch operations

db.batch_set({"a": 1, "b": 2, "c": [3, 4]})   # atomic group write
db.batch_create({"x": 10, "y": 20})            # all-or-nothing create
db.batch_delete(["a", "b"])                     # returns count deleted

---

Namespaces

Namespaces are zero-cost logical sub-spaces — keys are stored as <ns>\x00<key>.

users = db.ns("users")
posts = db.ns("posts")

users.set("alice", {"age": 30})
posts.set("p1",    {"title": "Hello"})

users.get("alice")   # {'age': 30}
posts.get("alice")   # None — isolated

users.keys()         # ['alice']
users.count()        # 1
users.clear()        # delete all keys in namespace

---

Query

# All keys starting with a prefix
db.prefix_scan("user:")                   # [(key_bytes, value), ...]

# Lexicographic range
db.range_scan("user:0050", "user:0099")

# Namespace sub-prefix
ns = db.ns("events")
ns.prefix_scan("2024:")

---

Snapshots

Snapshots capture an immutable read view. Writes after the snapshot are invisible to it.

snap = db.snapshot()
print(len(snap))            # key count at capture time

val = snap.read(b"key")     # raw bytes
for k, v in snap.items():   # iterate (yields key_bytes, raw_bytes)
    ...

# Snapshot is lightweight — just a dict copy + shallow segment refs.
# Release it before calling compact().
del snap

---

Durability modes

from lxcore import DURABILITY_UNSAFE, DURABILITY_NORMAL, DURABILITY_STRICT

db = LxDB("mydb", durability=DURABILITY_UNSAFE)   # no fsync — max throughput
db = LxDB("mydb", durability=DURABILITY_NORMAL)   # periodic fsync (default, every 5s)
db = LxDB("mydb", durability=DURABILITY_STRICT)   # fsync on every write

Mode	Throughput	Durability guarantee
unsafe	~340 K ops/s	data may be lost on power failure
normal	~325 K ops/s	at most 5s of writes lost
strict	~470 ops/s	every write survives power failure

---

Compression

db = LxDB("mydb", compress=True)   # zlib level-1, transparent on read

Values are only compressed if the compressed form is smaller. Incompressible values are stored raw. No change to the API.

---

Compaction

Dead records (tombstones, overwritten values) accumulate in segments. Compaction rewrites all live records into a single segment and drops the rest.

# Manual
db.compact()

# Automatic (default: runs when dead_ratio > 40%)
db = LxDB("mydb", auto_compact=True, compact_threshold=0.4, compact_interval=30.0)

---

Checkpoints

On close, lxcore saves a binary index checkpoint (checkpoint.lxidx). On the next open, the checkpoint is loaded and only records newer than it are replayed — making startup fast even on large databases.

db.checkpoint()   # force checkpoint at any time

---

Metrics & introspection

db.metrics()
# {
#   'key_count': 50000, 'segment_count': 3,
#   'total_bytes': 8_400_000, 'dead_bytes': 210_000, 'dead_ratio': 0.025,
#   'writes': 50000, 'reads': 12000, 'deletes': 500,
#   'ops_per_sec': 183000.0, 'cache_hit_ratio': 0.94,
#   'compact_count': 1, 'secs_since_compact': 12.4,
#   ...
# }

db.info()   # full dump including segment map, writer_id, durability mode

---

LRU value cache

db = LxDB("mydb", max_cache=4096)   # cache last 4096 values (default: 2048)
db = LxDB("mydb", max_cache=0)      # disable cache

---

Segment configuration

db = LxDB("mydb", max_segment_size=128 * 1024 * 1024)   # 128 MB per segment (default: 64 MB)

When a segment reaches max_segment_size, a new one is created automatically. Compaction collapses all segments back to one.

---

Migration from v1

If you have a legacy single-file .lxdb database (lxcore < 1.0):

from lxcore import migrate_v1

engine = migrate_v1("/path/to/old.lxdb", "/path/to/new")
engine.close()

---

Database viewer

lxcore ships a full-featured web viewer built on Flask:

python viewer/app.py /path/to/mydb --port 5000

Open http://localhost:5000 in your browser. Features:

• Browse — paginated key/value table with search, prefix filter, inline edit, delete
• Segments — visual size bars per segment file
• Metrics — live dashboard (key count, ops/sec, dead ratio, cache hit rate, …)
• Raw Scanner — low-level record-by-record view with CRC status
• Tools — compact, checkpoint, flush, v1 migration — all one click

---

Stress test

python tests/stress.py             # 50 000 ops (default)
python tests/stress.py 200000      # custom scale

Covers 18 suites: raw CRUD, typed API, batch ops, namespaces, prefix/range scans, snapshots, compaction, checkpoint reload, compression, 3 durability modes, 8-thread concurrent reads, concurrent write+read, LRU cache, metrics, CRC crash consistency, segment rotation, v1 migration, and 1 MB value roundtrip.

---

Architecture reference

mydb.lxdb/
├── seg_00000001.lx       # append-only segment (14-byte header + records)
├── seg_00000002.lx       # created when previous segment fills
└── checkpoint.lxidx      # persistent index (binary, ~23B + 18B per key)

Record format (v2)

STATUS (1B) FLAGS (1B) SEQ (8B) KEY_LEN (2B) VAL_LEN (4B)  →  16 bytes header
KEY   (KEY_LEN bytes)
VALUE (VAL_LEN bytes)
CRC32 (4 bytes, over all of the above)

FLAGS bit-0 = value is zlib-compressed.
SEQ is a monotonic uint64 writer sequence number used for checkpoint replay.

---

Benchmark (single machine, Python 3.12)

Operation	Throughput
batch_delete	582 K ops/s
raw_delete	390 K ops/s
full_scan_reload	430 K ops/s
checkpoint_reload	342 K ops/s
raw_create	284 K ops/s
raw_read	256 K ops/s
batch_set	212 K ops/s
typed_set	124 K ops/s
typed_get	162 K ops/s
compact (50 K records)	329 K ops/s
concurrent reads (8 threads)	17 K ops/s

---

Changelog

0.2.0

• Segment-based append-only storage (replaces single file)
• Persistent index checkpoints for fast startup
• Background compactor daemon
• Snapshot isolation (immutable read views)
• LRU value cache
• Tunable durability modes (unsafe / normal / strict)
• Optional zlib value compression
• Prefix and range scan queries
• Sequence numbers on every record
• Runtime metrics & introspection
• v1 single-file migration utility
• Web-based database viewer (Flask)
• Fast native int/float serialization paths

0.1.0

• Initial release

---

License

MIT — see LICENSE.