AethisDB

A lightweight, pure-Python document database with a custom binary file format (.aethisdb). Zero external dependencies — uses only the Python standard library.

Features

• Custom binary file format with a 64-byte header and append-friendly data section
• Hash-based indexing for fast exact-match queries on indexed fields
• Full CRUD operations: insert, find, update, delete
• Rich query operators: $eq, $ne, $gt, $lt, $gte, $lte, $in, $nin, $exists, $and, $or
• Auto-commit or manual commit modes
• File compaction to reclaim space from deleted records
• Automatic compaction based on tombstone ratio — ideal for small/embedded storage
• zlib-compressed index block for minimal on-disk footprint
• Compact JSON serialization (no whitespace in record data)
• Space statistics reporting via db.stats()
• Context manager support (with Database(...) as db:)
• Persistent across sessions — collections and indexes survive reopens

Installation / Setup

AethisDB has no third-party dependencies. Simply place the aethisdb/ package directory on your Python path, or install it in development mode:

pip install aethisdb

Then import:

from aethisdb import Database, Record

Quick Start

Initializing a Database

from aethisdb import Database

# Creates a new .aethisdb file (or opens an existing one)
db = Database("mydata.aethisdb")

# Create a collection
users = db.create_collection("users")

# List collections
print(db.list_collections())  # ['users']

Inserting Records

# Insert a single record — returns the auto-assigned record ID
uid = users.insert({"name": "Alice", "email": "alice@example.com", "age": 30})

# Insert multiple records at once
ids = users.insert_many([
    {"name": "Bob",   "email": "bob@example.com",   "age": 25},
    {"name": "Carol", "email": "carol@example.com", "age": 35},
])

# Get a record by ID
alice = users.get(0)
print(alice["name"])  # Alice

Querying Records

# Find by exact match
results = users.find({"name": "Alice"})
print(results[0]["email"])  # alice@example.com

# Find one (returns None if no match)
bob = users.find_one({"email": "bob@example.com"})

# Range query
adults = users.find({"age": {"$gte": 30}})

# Multiple conditions (implicit AND)
results = users.find({"age": {"$gte": 25}, "age": {"$lte": 35}})

# $in operator
results = users.find({"name": {"$in": ["Alice", "Bob"]}})

# $or operator
results = users.find({"$or": [{"age": 25}, {"age": 35}]})

# Get all records
all_users = users.all()

# Count records
print(users.count())  # 3

Updating Records

# Update matching records
count = users.update({"name": "Alice"}, {"age": 31})
print(count)  # 1

# Update by record ID
users.update(record_id=1, update_data={"age": 26})

Note: Updates in AethisDB use a copy-on-write strategy — the old record is marked as deleted and a new record is appended with the updated data. Record IDs are preserved.

Deleting Records

# Delete by query
count = users.delete({"name": "Bob"})

# Delete by record ID
count = users.delete(record_id=2)

# Delete all records (use with care!)
users.delete()

# Count after deletion
print(users.count())  # 2

Indexing

Hash indexes provide O(1) exact-match lookups:

# Create an index on a field
users.create_index("email")

# Now queries on the indexed field use the index automatically
bob = users.find_one({"email": "bob@example.com"})  # fast

# List indexes
print(users.indexes)  # ['email']

# Drop an index
users.drop_index("email")

# Rebuild an index after bulk operations
users.create_index("email")
users.rebuild_index("email")

Compaction

Deleted records leave tombstones in the file. Use compact() to reclaim space:

# After many deletions
file_size_before = os.path.getsize("mydata.aethisdb")

db.compact()

file_size_after = os.path.getsize("mydata.aethisdb")
# File is smaller — tombstones removed

Transaction / Auto-commit Control

# Manual commit mode — changes are not persisted until commit()
db = Database("mydata.aethisdb", auto_commit=False)

users = db.get_collection("users")
users.insert({"name": "Alice"})
users.insert({"name": "Bob"})

# Verify data is not yet persisted
db.close()  # Without commit, changes are lost

# Correct usage:
db = Database("mydata.aethisdb", auto_commit=False)
users = db.get_collection("users")
users.insert({"name": "Alice"})
users.insert({"name": "Bob"})
db.commit()  # persist to disk
db.close()

Context Manager

with Database("mydata.aethisdb") as db:
    users = db.get_collection("users")
    users.insert({"name": "Alice"})
    alice = users.find_one({"name": "Alice"})
# db is automatically closed

Space Efficiency (Small Storage)

AethisDB includes several features to minimize disk usage, making it ideal for embedded or small-storage applications:

Compact JSON

All record data and the index block use compact JSON serialization (no whitespace, no separator padding). This reduces file size compared to standard json.dumps().

Compressed Index Block

The index block at the end of the file is automatically zlib-compressed when compression reduces its size. Compression is transparent — no configuration required.

Auto-Compaction

When records are deleted, they are soft-deleted (marked as tombstones). Over time, tombstones waste space. AethisDB can automatically compact the file on close when the tombstone ratio exceeds a configurable threshold:

from aethisdb import Database

# auto_compact=True: compact on close if deleted ratio > compact_ratio
# compact_ratio=0.3 (default): compact when 30%+ of records are deleted
db = Database("small.aethisdb", auto_compact=True, compact_ratio=0.25)

users = db.create_collection("users")
for i in range(10):
    users.insert({"name": f"user_{i}"})
for i in range(5):
    users.delete({"name": f"user_{i}"})
# 50% deleted > 25% threshold → file is compacted automatically on close
db.close()

Manual Commit Mode

For workloads with many writes, use manual commit mode to reduce index write frequency:

# Only write the index to disk when commit() or close() is called
db = Database("small.aethisdb", auto_commit=False)

users = db.create_collection("users")
for i in range(1000):
    users.insert({"name": f"user_{i}"})

db.commit()  # one index write instead of 1000
db.close()

Space Statistics

db = Database("small.aethisdb")
stats = db.stats()
print(stats)
# {
#   'file_size': 4384,
#   'header_size': 64,
#   'data_size': 3415,
#   'index_size': 905,
#   'record_count': 50,
#   'deleted_count': 0,
#   'deleted_bytes_estimate': 0,
#   'collections': 1,
#   'auto_compact': False,
#   'auto_commit': True
# }
db.close()

Manual Compaction

Force a compaction at any time:

db = Database("small.aethisdb")
users = db.get_collection("users")
users.delete({"age": {"$lt": 18}})
db.compact()  # immediately rewrite file, removing tombstones
db.close()

Full Example

from aethisdb import Database

with Database("company.aethisdb") as db:
    employees = db.create_collection("employees")
    employees.create_index("email")

    # Seed data
    employees.insert_many([
        {"name": "Alice",   "email": "alice@corp.com", "department": "Engineering", "salary": 90000},
        {"name": "Bob",     "email": "bob@corp.com",   "department": "Sales",       "salary": 65000},
        {"name": "Charlie", "email": "carol@corp.com", "department": "Engineering", "salary": 85000},
    ])

    # Query with index
    alice = employees.find_one({"email": "alice@corp.com"})

    # Range query
    high_earners = employees.find({"salary": {"$gte": 80000}})

    # Update
    employees.update({"email": "bob@corp.com"}, {"salary": 70000})

    # Delete
    employees.delete({"department": "Sales"})

    # Verify
    remaining = sorted(e["name"] for e in employees.all())
    print(remaining)  # ['Alice', 'Charlie']

---

File Format: .aethisdb

Overview

The .aethisdb file format is a custom binary layout designed for append-friendly record storage with a persistent JSON index at the end of the file. The format consists of three sections:

┌──────────────────┬──────────────────┬─────────────────────┐
│   Header (64B)   │  Data Section    │  Index Block (JSON) │
│  (fixed size)    │ (append-only)    │   (end of file)     │
├──────────────────┼──────────────────┼─────────────────────┤
│                  │ Collection       │ {                   │
│  Magic number    │   Markers        │   "version": 2,     │
│  Version         │ Data records     │   "collections": {  │
│  Flags           │                  │     "users": {      │
│  Header size     │                  │       ...           │
│  Index offset    │                  │     }               │
│  Index length    │                  │   }                 │
│  Data start      │                  │ }                   │
│  Collection count│                  │                     │
│  Reserved        │                  │                     │
└──────────────────┴──────────────────┴─────────────────────┘

1. Header (exactly 64 bytes)

The header is a fixed-size binary structure at the beginning of the file:

Offset	Size	Type	Field	Description
0	8	8s	Magic number	Always b"AETHISDB"
8	1	uint8	Version	Format version (currently 2)
9	1	uint8	Flags	Reserved flags (currently 0)
10	2	uint16	Header size	Always 64
12	4	uint32	Reserved	Zero
16	8	uint64	Index offset	Byte offset to the index block
24	8	uint64	Index length	Size of the index block in bytes
32	8	uint64	Data start	Always 64 (where data begins)
40	8	uint64	Collection count	Number of collections
48	16	16s	Reserved	Zero-padded

All integers are little-endian (struct format character <).

2. Data Section

The data section starts at byte offset 64 (right after the header). It contains a sequence of block entries. Two block types are supported:

Collection Marker Block

Written when a collection is created. Identifies the start of a collection's data.

Offset	Size	Type	Field	Description
0	1	uint8	Type	1 (COLLECTION_MARKER_TYPE)
1	1	uint8	Name length	Length of collection name (bytes)
2	var	bytes	Name	UTF-8 encoded collection name
2+n	8	uint64	Next ID	Next auto-increment record ID

Record Entry Block

Written for each record insert. Records are appended sequentially.

Offset	Size	Type	Field	Description
0	1	uint8	Type	2 (RECORD_ENTRY_TYPE)
1	1	uint8	Status	0 = deleted, 1 = active
2	8	uint64	Record ID	Unique record identifier
10	4	uint32	Data length	Length of JSON data (bytes)
14	var	bytes	Data	JSON-encoded record payload

3. Index Block (JSON, end of file)

The index block is a JSON document appended after the last data byte. The header's index_offset and index_length fields locate it. On every commit, the entire index is rewritten.

Structure:

{
  "version": 2,
  "collections": {
    "users": {
      "name": "users",
      "marker_offset": 64,
      "next_id": 3,
      "record_count": 2,
      "deleted_count": 1,
      "record_positions": {
        "0": 92,
        "1": 130
      },
      "indexes": {
        "email": {
          "type": "hash",
          "field": "email",
          "values": {
            "alice@example.com": [0],
            "bob@example.com": [1]
          }
        }
      }
    }
  }
}

Key fields:

Field	Type	Description
marker_offset	integer	Byte offset of the collection marker in the data section
next_id	integer	Next auto-increment record ID
record_count	integer	Number of active (non-deleted) records
deleted_count	integer	Number of soft-deleted records
record_positions	object	Maps record_id (as string) to byte offset in file
indexes	object	Hash indexes for indexed fields

How Writes Work

AethisDB uses an append-friendly strategy:

1. Insert: A record is appended to the end of the data section. Its position is recorded in record_positions.
2. Update: The old record's status byte is set to 0 (deleted) in-place. A new record with the updated data is appended at the end. The record_positions map is updated to point to the new record. The old record remains as a tombstone.
3. Delete: The record's status byte is set to 0 (deleted) in-place. It is removed from record_positions and from all hash indexes.
4. Commit: The entire index block is serialized as JSON and written after the last data byte. The 64-byte header is updated with the new index_offset and index_length.

How Reads Work

1. The header is read (first 64 bytes) to locate the index block.
2. The index JSON is parsed to reconstruct collection metadata and hash indexes in memory.
3. For single-record lookups (get(id)), the byte offset is retrieved from record_positions, the file is seeked to that position, and the record is read.
4. For indexed exact-match queries, the hash index directly maps field values to record IDs, avoiding any file scanning.
5. For non-indexed or range queries, all active records are scanned and filtered using the query engine.

How Compaction Works

When compact() is called:

1. A temporary file is created.
2. All active records (status = 1) are read from the current file and rewritten sequentially to the temp file.
3. Deleted records (tombstones) are skipped — they are not copied.
4. A fresh, compact index is written at the end of the temp file.
5. The temp file replaces the original file atomically (os.replace).
6. The result is a file with no wasted space from tombstones or orphaned index blocks.

Query Operators

Operator	Type	Description	Example
(default)	Exact match	Field equals value	{"name": "Alice"}
$eq	Equality	Field equals value	{"age": {"$eq": 30}}
$ne	Inequality	Field does not equal value	{"age": {"$ne": 30}}
$gt	Comparison	Field greater than value	{"age": {"$gt": 25}}
$lt	Comparison	Field less than value	{"age": {"$lt": 35}}
$gte	Comparison	Field greater than or equal to value	{"age": {"$gte": 30}}
$lte	Comparison	Field less than or equal to value	{"age": {"$lte": 30}}
$in	Set	Field value is in list	{"role": {"$in": ["a", "b"]}}
$nin	Set	Field value is not in list	{"role": {"$nin": ["a"]}}
$exists	Existence	Field exists (true) or doesn't exist (false)	{"email": {"$exists": true}}
$and	Logical	All sub-conditions must match	{"$and": [cond1, cond2]}
$or	Logical	At least one sub-condition must match	{"$or": [cond1, cond2]}

Running Tests

cd /path/to/aethisdb
python -m unittest discover -s tests -v

Architecture

aethisdb/
├── __init__.py      Package exports (Database, Collection, Record, etc.)
├── database.py      Top-level Database class — file lifecycle & collection mgmt
├── collection.py    Collection class — CRUD operations & query optimization
├── storage.py       StorageEngine — binary file format I/O (header, records, index)
├── index.py         Hash-based Index & IndexManager for fast lookups
├── record.py        Record class — dict-like document with an ID
├── query.py         Query engine — condition parsing, evaluation, update application
├── utils.py         Constants and struct helpers for the binary format
└── exceptions.py    Custom exception hierarchy