lqft-python-engine 0.8.7

LQFT Engine: Zero-Copy Buffer Protocol & Hardware Saturation (v0.8.7 Stable)

Log-Quantum Fractal Tree (LQFT) 🚀

📌 Project Overview

The Log-Quantum Fractal Tree (LQFT) is a high-performance, scale-invariant data structure engine designed for massive data deduplication and persistent state management. It synthesizes Hash Array Mapped Trie (HAMT) routing with Merkle-DAG structural folding to provide deterministic $O(1)$ search latency and sub-linear $O(\Sigma)$ space complexity.

By offloading core associative logic to a native C-Extension, the LQFT completely bypasses the Python Global Interpreter Lock (GIL), providing true hardware concurrency and significant memory reduction for versioned, redundant, or patterned datasets.

---

🧠 Core Architecture (v0.8.1 Enterprise Release)

1. Zero-Copy Buffer Protocol (New in v0.8.1)

The engine now features insert_batch_raw, a low-level C-API endpoint that accepts contiguous memory buffers (like Python's native array). This bypasses the heavy PyLong object conversion overhead, allowing the engine to ingest data at the absolute physical limit of the CPU's memory bus.

2. True Hardware Concurrency & Strict GIL Bypass

The engine utilizes native OS-level read-write locks (SRWLOCK on Windows, pthread_rwlock_t on POSIX) combined with strict Py_BEGIN_ALLOW_THREADS boundaries.

• Multi-Core Scaling: Multiple Python threads can read and write to the Merkle-DAG simultaneously across all physical CPU cores without Segmentation Faults.
• Zero GIL Contention: The C-Engine entirely decouples from the Python interpreter during execution.

3. Scale-Invariant Time Complexity: $O(1)$

The LQFT utilizes a fixed 64-bit hash space partitioned into 13 levels.

• Deterministic Latency: Every search or insertion requires exactly 13 pointer hops.
• Scale-Invariance: Performance remains constant whether the dataset contains 10^3 or 10^9 items.

4. Entropy-Based Space Complexity: $O(\Sigma)$

Nodes are identified by the cryptographic hash of their contents and child pointers (Merkle-DAG).

• Structural Folding: Identical sub-trees are shared physically in memory across different branches or versions.

---

🚀 Performance Benchmarks (Hardware Saturation Reached)

Environment: Python 3.12 | GCC -O3 | MSYS2/MinGW64 | 16-Thread Concurrency

Metric	Result
Read Throughput (16-Core)	~36.02 Million ops/sec
Write Throughput (Zero-Copy)	~320,000 ops/sec (Saturates DDR RAM latency at ~19.5M internal memory jumps/sec)
Space Efficiency	Up to 1,500x reduction in versioned graph simulations
Stability	100% Memory Safe under massive multi-threaded turnover

---

🛠️ Getting Started

Installation

The engine requires a C compiler (GCC/MinGW or MSVC) to build the native extension.

# Clone the repository
git clone [https://github.com/ParjadM/Log-Quantum-Fractal-Tree-LQFT-.git](https://github.com/ParjadM/Log-Quantum-Fractal-Tree-LQFT-.git)
cd Log-Quantum-Fractal-Tree-LQFT-

# Build the native C-extension
python setup.py build_ext --inplace

High-Performance Usage (Zero-Copy Buffer)

For extreme ingestion tasks, bypass standard Python objects and feed the C-Engine directly from RAM.

import lqft_c_engine
import array
import hashlib

# 1. Prepare raw 64-bit integer buffer
raw_buffer = array.array('Q') 
for i in range(100000):
    h = int(hashlib.md5(f"data_{i}".encode()).hexdigest()[:16], 16)
    raw_buffer.append(h)

# 2. Ingest at Silicon Speeds (Zero Python Overhead)
lqft_c_engine.insert_batch_raw(bytes(raw_buffer), "enterprise_payload")

# 3. Read at Hardware Speeds
result = lqft_c_engine.search(raw_buffer[0])
print(result) # "enterprise_payload"

# 4. Native Disk Persistence
lqft_c_engine.save_to_disk("production_state.bin")
lqft_c_engine.free_all()
lqft_c_engine.load_from_disk("production_state.bin")

---

⚖️ License

This project is licensed under the MIT License - see the LICENSE.md file for details.