Nemo

Nemo is an ultra-high-performance, embedded, local relational database engine engineered in modern ISO C++20 with zero-copy Python bindings. Designed from the ground up as a high-concurrency alternative to SQLite, Nemo eliminates database-level lock contention, enabling simultaneous readers and simultaneous writers without deadlocks or database is locked collisions.

Nemo persists data into local .nm binary files, natively supports 22+ PostgreSQL-inspired datatypes (including JSONB, RANGE, INET, CIDR, BIT, UUID, ARRAY, DICT, and ENUM), provides over 70 built-in SQL scalar and extraction functions, supports database-managed sequences with atomic ID range allocation, features auto-increment primary keys, and exposes a unified synchronous and asynchronous (asyncio) Python SDK.

Table of Contents

• Why Nemo?
• How Nemo Works (Internal Architecture)
• Key Features
• Performance & Benchmarks (vs SQLite)
• Supported Datatypes
• Installation
• Quick Start
  • 1. Synchronous Usage
  • 2. Asynchronous Usage
  • 3. Built-in SQL Functions & Extraction
  • 4. Sequences & Atomic Range Allocation
  • 5. Atomic FIFO & LIFO Popping
• Comprehensive Documentation Index
• License

Why Nemo?

Traditional embedded databases like SQLite rely on database-wide serialization locks. Even in Write-Ahead Logging (WAL) mode, SQLite restricts writes to a single writer at a time. Under multi-threaded workers, background tasks, or async web servers, concurrent writes frequently trigger sqlite3.OperationalError: database is locked.

Nemo solves this concurrency bottleneck:

• Zero-Lock Concurrency: Multiple writers and multiple readers operate concurrently across tables without database lock contention.
• Extreme Write Throughput: Over 1,000,000+ rows/sec bulk ingestion and over 400,000 rows/sec single-row inserts.
• Rich Data Modeling: 22+ native datatypes including JSON, JSONB, Arrays, Key-Value Dicts, Ranges, INET/CIDR, Decimals, and UUIDs.
• 70+ Inbuilt SQL Functions: Complete library for document manipulation, vector math, text slicing, temporal parts, and binary encoding.
• Atomic Queue / Stack Primitives: Native pop() directly inside the C++ engine executing at over 8.5 Million operations/second.
• Dual SDK Surfaces: Unified synchronous connector (connect_sync()) and asynchronous connector (connect_async()).

How Nemo Works (Internal Architecture)

Nemo is engineered from scratch in ISO C++20 as an embedded, in-process database engine operating directly inside the application process with zero network overhead.

1. Slotted Page Storage & Disk Layout

Nemo stores database pages in uniform 4096-byte slotted blocks:

• Page Header: 64-byte metadata block storing Page ID, LSN (Log Sequence Number), transaction visibility flags, and CRC32 checksums.
• Slot Directory: Dynamic array of offset pointers growing forward from the page header.
• Record Storage: Variable-length and fixed-length records packed from the bottom of the page upward to eliminate fragmentation.

2. Multi-Version Concurrency Control (MVCC)

• Snapshot Isolation: When a transaction starts, it reads against a consistent snapshot of the database state.
• Lock-Free Reads: Readers never block writers, and writers never block readers.
• In-Memory Undo Chains: Active transactions maintain private undo records. Aborted transactions revert changes in microseconds without disk rewrites.

3. Concurrency & CLOCK Buffer Management

• Buffer Pool: Configurable multi-frame page cache managing pinned memory pages.
• CLOCK Eviction: Second-chance CLOCK replacement policy ensuring hot pages remain resident while evicting cold pages under memory pressure.
• Fine-Grained Latching: Table-level and page-level reader-writer shared mutexes replace database-level global locks.

4. Write-Ahead Logging (WAL) & Crash Recovery

• Segmented WAL Files: Write-ahead records stream to segmented .nm-wal logs before dirty pages flush to the main .nm file.
• Instant Recovery: On startup after an unexpected crash, committed transactions are replayed from the WAL and uncommitted transactions are rolled back.

Key Features

• Embedded & Serverless: Zero configuration, no external daemon, and zero network latency. Operates directly on .nm local binary files.
• True Concurrency: Concurrent multi-reader and multi-writer execution using fine-grained latching and MVCC snapshot isolation.
• Dual Ingestion Paths:
  • Vectorized Bulk Streaming (executemany): Rapid streaming ingestion for massive batch workloads.
  • Single Writing (execute): Ultra-low latency single-row inserts for real-time transaction processing.
• Auto-Increment & Sequences: Monotonically increasing primary keys and atomic range reservation (allocate_id_range).
• High-Velocity Queues: Native FIFO queue and LIFO stack popping (conn.pop(table, count, direction)).
• Fast Table Maintenance: Instant table row counting (conn.count(table)) and instant truncation (conn.truncate(table)).
• ACID Reliability: Full transaction support with Write-Ahead Logging (WAL) and rollback guarantees.

Performance & Benchmarks (vs SQLite)

Nemo was benchmarked against SQLite 3 (configured with PRAGMA journal_mode = WAL and PRAGMA synchronous = OFF) on identical datasets of 500,000 bulk records and 25,000 single-row writes across 5 heterogeneous multi-format tables with 16 simultaneous concurrent threads:

For deep architectural details on how Nemo achieves these metrics, read the Engine Architecture & How It Works Guide.

Supported Datatypes (30 Native Types)

Nemo features a comprehensive, PostgreSQL-inspired type registry with native support for 30 strongly typed datatypes:

Installation

Install the package from PyPI:

pip install nemo-py

Once installed, import the engine as nemo:

import nemo

Quick Start

1. Synchronous Usage

import nemo

# Connect using database name (no extension required)
with nemo.connect("production") as db:
    with db.connect_sync() as conn:
        conn.execute("""
            CREATE TABLE accounts (
                id BIGINT PRIMARY KEY AUTOINCREMENT,
                username VARCHAR NOT NULL,
                balance DOUBLE DEFAULT 0.0,
                is_active BOOLEAN DEFAULT true,
                metadata JSON,
                tags ARRAY
            );
        """)

        # Insert record with parameterized values
        conn.execute(
            "INSERT INTO accounts (username, balance, metadata, tags) VALUES (?, ?, ?, ?);",
            ("alice", 1500.50, {"tier": "gold", "role": "admin"}, ["vip", "crypto"])
        )

        # Query records
        result = conn.execute("SELECT * FROM accounts WHERE username = ?;", ("alice",))
        row = result.fetchone()
        print("Fetched account:", row)

2. Asynchronous Usage

import asyncio
import nemo

async def main():
    with nemo.connect("production") as db:
        async with db.connect_async() as conn:
            result = await conn.execute("SELECT username, balance FROM accounts;")
            async for username, balance in result:
                print(f"User: {username}, Balance: {balance}")

asyncio.run(main())

3. Built-in SQL Functions & Extraction

with nemo.connect("production") as db:
    with db.connect_sync() as conn:
        # JSON & JSONB Extraction
        res = conn.execute("SELECT json_extract(metadata, '$.role') FROM accounts WHERE id = 1;")
        print("Role:", res.fetchone()[0])  # 'admin'

        # ARRAY functions
        res = conn.execute("SELECT array_contains(tags, 'vip'), array_length(tags) FROM accounts WHERE id = 1;")
        print("Tags stats:", res.fetchone())  # (True, 2)

        # System & Temporal functions
        res = conn.execute("SELECT now(), uuid_generate();")
        print("Current time & UUID:", res.fetchone())

4. Sequences & Atomic Range Allocation

with nemo.connect("production") as db:
    with db.connect_sync() as conn:
        # Create sequence
        conn.execute("CREATE SEQUENCE order_seq START WITH 1000 INCREMENT BY 5;")

        # SQL nextval
        new_id = conn.execute("SELECT nextval('order_seq');").fetchone()[0]

        # Atomic range allocation for high-speed worker threads
        start_id, end_id = conn.allocate_id_range("order_seq", count=50)
        print(f"Allocated block: {start_id} to {end_id}")

5. Atomic FIFO & LIFO Popping

with nemo.connect("production") as db:
    with db.connect_sync() as conn:
        # Atomically pop 10 items from head (FIFO queue)
        jobs = conn.pop("job_queue", count=10, direction="fifo")

        # Atomically pop 5 items from tail (LIFO stack)
        recent_events = conn.pop("event_log", count=5, direction="lifo")

Comprehensive Documentation Index

All technical documentation is organized in the docs/ directory and root architecture specifications:

• Engine Architecture & How It Works: Deep-level architectural breakdown of the execution lifecycle, 4KB slotted page format, buffer pool CLOCK eviction, fine-grained table latching, CBO optimizer, and crash recovery.
• SQL Built-in Functions Reference: Exhaustive reference for 70+ built-in functions (JSON, JSONB, ARRAY, DICT, String, Math, Range, Temporal, Binary, Bitwise, Network, Sequences).
• SQL Parameters & Prepared Statements Reference: Complete guide to parameter placeholders (?), type coercion, batch execution, prepared statements, and security redaction.
• Nemo Python Functions & API Reference: Complete Python API reference for Database, SyncConnection, AsyncConnection, Result, PreparedStatement, and connection pools.
• Datatypes Reference: Complete specification of all 30 native datatypes, storage bounds, and Python type mapping.
• SQL Syntax & Query Reference: DDL, DML, constraints, auto-increment primary keys, sequences, CBO optimizer, hash joins, SIMD aggregates, window functions, and native table operations (pop, count, truncate).
• Connecting to Nemo: Detailed guide to path resolution, timeout handling, connection pooling, and lifecycle management.
• Synchronous Python Connector: Full reference for synchronous connections, statements, columnar extraction, and cursor management.
• Asynchronous Python Connector: Full reference for asyncio execution, async cursors, and event loop integration.
• Transactions & Concurrency Guide: MVCC snapshot isolation, transaction patterns, autocommit modes, and concurrency benchmarks.
• Logging, Diagnostics & Error Hierarchy: Query logging configuration, slow query thresholds, parameter masking, and exception trees.
• Nemo Python Examples Suite (50 Standalone Scripts): Master collection of 50 runnable, self-contained examples covering every operational aspect, datatype, SQL function, and concurrency pattern.
• Peak Performance & Concurrency Benchmarks: Reproducible benchmark suite comparing Nemo with SQLite across 18 key metrics.

License

Nemo is licensed under the Apache License, Version 2.0.
Copyright 2026 Pravanjan Roy (kingmon6996).

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at:

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the LICENSE file for the specific language governing permissions and limitations under the License.