sop4py 2.0.16

Scalable Objects Persistence (SOP) V2 for Python. General Public Availability (GPA) Release

SOP for Python (sop4py)

Scalable Objects Persistence (SOP) is a high-performance, transactional storage engine for Python, powered by a robust Go backend. It combines the raw speed of direct disk I/O with the reliability of ACID transactions and the flexibility of modern AI data management.

Key Features

• Transactional B-Tree Store: Unlimited, persistent B-Tree storage for key-value data.
• Vector Database: Built-in vector search (k-NN) for AI embeddings and similarity search.
• AI Model Store: Versioned storage for machine learning models (B-Tree backed).
• ACID Compliance: Full transaction support (Begin, Commit, Rollback) with isolation.
• High Performance: Written in Go with a lightweight Python wrapper (ctypes).
• Caching: Integrated Redis-backed L1/L2 caching for speed.
• Replication: Optional Erasure Coding (EC) for fault-tolerant storage across drives.
• Flexible Deployment: Supports both Standalone (local) and Clustered (distributed) modes.

Prerequisites

• Redis: Required for caching and transaction coordination (especially in Clustered mode).
• Storage: Local disk space (supports multiple drives/folders).
• OS: macOS (Darwin), Linux, or Windows (AMD64).

Installation

pip install sop4py

Note: Ensure the libjsondb shared library is available in your library path if building from source.

Quick Start Guide

1. Setup & Configuration

First, import the necessary modules and configure your storage environment.

from sop import Context, Transaction, TransactionOptions, TransactionMode

# Initialize Context
ctx = Context()

# Configure Transaction Options
opts = TransactionOptions(
    mode=TransactionMode.ForWriting.value,
    max_time=15,  # 15 minutes timeout
    registry_hash_mod=250,
    stores_folders=["/tmp/sop_data"],  # Path to store data
    erasure_config={}  # Optional replication config
)

2. Transactional Key-Value Store (B-Tree)

SOP allows you to manage data using B-Trees within a transaction.

from sop import btree

# Use the Transaction Context Manager
with Transaction(ctx, opts) as t:
    # Create or Open a B-Tree
    # "users" is the store name, True indicates native key type
    bo = btree.BtreeOptions("users", True, cache_config=btree.CacheConfig())
    store = btree.Btree.new(ctx, bo, t)

    # Add Items
    store.add(ctx, [
        btree.Item(key=1, value="Alice"),
        btree.Item(key=2, value="Bob")
    ])

    # Fetch Items
    item = store.find_one(ctx, 1, False)
    print(f"Found: {item.value}")

# Transaction is automatically committed here

3. Vector Database (AI)

Manage vector embeddings for semantic search. You can choose between Standalone (local) and Clustered (distributed) modes.

from sop.ai import VectorDatabase, UsageMode, Item, DBType

# Initialize Vector DB
# DBType.Standalone: Uses in-memory caching (good for single node).
# DBType.Clustered: Uses Redis for caching (good for distributed setups).
vdb = VectorDatabase(
    storage_path="/tmp/sop_vectors", 
    usage_mode=UsageMode.Dynamic,
    db_type=DBType.Standalone
)
vector_store = vdb.open("products")

# Upsert Vectors
vector_store.upsert(Item(
    id="prod_101",
    vector=[0.1, 0.5, 0.9],
    payload={"name": "Laptop", "price": 999}
))

# Semantic Search (k-NN)
hits = vector_store.query(
    vector=[0.1, 0.5, 0.8],
    k=5,
    filter={"name": "Laptop"}
)

for hit in hits:
    print(f"Match: {hit.id}, Score: {hit.score}")

4. AI Model Store

Version and manage your machine learning models using the B-Tree backend.

from sop.ai import Database, Model

# Initialize the Database
db = Database(storage_path="/tmp/sop_data")

# Open the Model Store
model_store = db.open_model_store("my_models")

# Save a Model
model = Model(
    id="churn_v1",
    algorithm="random_forest",
    hyperparameters={"trees": 100},
    parameters=[0.5, 0.1, 0.9],  # Serialized weights
    metrics={"accuracy": 0.98},
    is_active=True
)
model_store.save("classifiers", "churn_v1", model)

# Retrieve a Model
loaded_model = model_store.get("classifiers", "churn_v1")
print(f"Loaded Model: {loaded_model['model']['id']}")

Examples

The examples/ directory contains complete, runnable demos showcasing various features of the library.

• vector_demo.py: Basic usage of the Vector Store in Standalone mode. Demonstrates auto-commit vs. explicit transactions.
• vector_clustered_demo.py: Usage of the Vector Store in Clustered mode with Redis caching.
• modelstore_demo.py: Managing AI models (save, load, list, delete) using the B-Tree backend.
• langchain_demo.py: Integration with LangChain workflows (mocked adapter example).
• vector_replication_demo.py: Advanced usage showing how to configure Erasure Coding (RAID-like redundancy) for vector data.

To run an example (assuming you are in the root of the repo):

PYTHONPATH=jsondb/python python3 jsondb/python/examples/vector_demo.py

AI Development Workflows

SOP is designed to support the full AI Software Development Life Cycle (SDLC), allowing you to transition seamlessly from local experimentation to distributed production.

Phase 1: Local Training & Development (Standalone)

Data Scientists and ML Engineers can work in Standalone Mode on their local machines. This mode isolates the environment, requiring no external dependencies like Redis.

• Action: Train models and populate vector stores locally.
• Configuration: Use DBType.Standalone.
• Benefit: Fast iteration, zero infrastructure overhead, full isolation.

# Local Development
vdb = VectorDatabase(storage_path="./local_vectors", db_type=DBType.Standalone)
# ... train model, generate embeddings, upsert to vdb ...

Phase 2: Production Deployment (Clustered)

Once the model and vector data are stabilized and validated, they can be "promoted" to the production environment.

• Action: Copy the storage directory (e.g., ./local_vectors) to the production shared storage volume.
• Configuration: Switch the application to DBType.Clustered.
• Benefit: The production cluster (backed by Redis) now serves the pre-computed data with high availability and caching.

# Production
vdb = VectorDatabase(storage_path="/mnt/prod_vectors", db_type=DBType.Clustered)
# ... serve queries ...

This workflow allows you to treat your AI data (models and vector indices) as artifacts that are built locally and released to production, ensuring consistency and simplifying the deployment pipeline.

Architecture

SOP uses a split architecture:

1. Core Engine (Go): Handles disk I/O, B-Tree algorithms, caching, and transactions. Compiled as a shared library (.dylib, .so, .dll).
2. Python Wrapper: Uses ctypes to interface with the Go engine, providing a Pythonic API (sop package).

Project Links

• Source Code: GitHub - sharedcode/sop
• PyPI: sop4py

Contributing

Contributions are welcome! Please check the CONTRIBUTING.md file in the repository for guidelines.