PROJECT LUMI

LuminaMoon (luminamoon) is the official reference implementation and central monorepo of PROJECT LUMI, providing low-precision numerical optimization, multimodal dataset streaming, and formal verification suites developed by Lumina Moon (@cloverx-id) and Silver Moon (@cloverxion).

Built with a modular architecture under the PROJECT LUMI specification, luminamoon connects specialized components for CPU/GPU hardware execution, low-latency streaming, and verified numerical stability:

System Architecture

flowchart TD
    Shards["Dataset Shards (Parquet / JSONL)"] --> DataLoader["LuminaMoon DataLoader (SIMD + XXH3 + Pre-RAM)"]
    DataLoader --> TrainLoop["Training Loop (Batch Stream)"]
    DataLoader --> Checkpoint["128B Ping-Pong Checkpoint & WAL"]
    TrainLoop --> Checkpoint
    TrainLoop --> Dispatcher["LuminaV Hardware Dispatcher"]
    Eval["LSO-Eval Suite (Z3 SMT & Parity Gate)"] -.-> Dispatcher
    Dispatcher --> Triton["Triton AOT/JIT GPU (CUDA / ROCm)"]
    Dispatcher --> Vulkan["Vulkan 1.3/1.4 Compute (UMA / SPIR-V)"]
    Dispatcher --> SIMD["Rayon CPU SIMD (AVX-512 / AVX2)"]
    Dispatcher --> Scalar["Reference Scalar Core (Fallback)"]

Projects & Components

Quickstart

Installation

Install the compiled package wheel from PyPI:

pip install luminamoon

Or build from source using Maturin and the Rust 2024 toolchain:

git clone https://github.com/cloverx-id/luminamoon.git
cd luminamoon
pip install maturin
maturin develop --release

Basic Usage (LuminaV Optimizer)

import numpy as np
from luminamoon import LuminaV

# Initialize parameters
weights = np.random.randn(64, 10).astype(np.float32)
biases = np.zeros(10, dtype=np.float32)

# Instantiate LuminaV optimizer
optimizer = LuminaV([weights, biases], lr=1e-3, buffer=2)

# Execute optimization step with explicit gradients
grad_w = np.random.randn(64, 10).astype(np.float32)
grad_b = np.random.randn(10, dtype=np.float32)
optimizer.step(grads=[grad_w, grad_b])

For PyTorch deep learning workflows, see the PyTorch Integration Guide.

Basic Usage (LuminaMoon DataLoader & Official HuggingFace Pipeline)

import luminamoon as lm

# 1. Stream local or remote HuggingFace datasets with O(1) instant resume,
#    100-batch PRE-RAM watermark prefetching, multi-core download, and XXH3-64 deduplication
for batch in lm.stream(
    "hf://datasets/org/corpus@main/train/part-000.parquet",
    batch_size=64,
    preram_batches=100,
    refill_trigger=10,
    download_workers=4,
    deduplicate=True,
):
    # batch.records contains SIMD/Rayon-cleaned strings; deferred auto_commit persists GPU state
    pass

# 2. Push multimodal tables (text, image bytes, JSON, numbers, booleans) to HuggingFace Hub
lm.push_to_hub(
    repo="cleaned-multimodal-dataset",
    data=[
        {"prompt": "Hello", "score": 0.99, "verified": True, "meta": {"lang": "id"}},
    ],
    path_in_repo="train/part-00000.parquet",
)

For multimodal Parquet streaming, datasets / transformers integration, dual-metadata crash recovery, and Colab/Kaggle secret resolution, see the DataLoader Guide.

Verification & CI/CD

All pull requests and releases are validated across Linux, Windows, and macOS:

# Enforce zero-warning static analysis across the Rust workspace
cargo clippy --workspace --all-targets -- -D warnings

# Run Rust workspace unit and differential parity tests (Optimizer + DataLoader)
cargo test --workspace

# Run Python static type check
npx pyright python/luminamoon

# Run standalone NumPy, PyTorch, and DataLoader verification suites
python optimizer/tests/test_standalone_numpy.py
python optimizer/tests/test_end_to_end_training.py
python dataloader/tests/test_dataloader_python.py

# Run LSO-Eval formal verification suite
python -m eval

Developer Resources

For contributors and systems engineers building on or extending PROJECT LUMI (luminamoon):

• Optimizer Modular Documentation: docs/optimize/
• DataLoader Modular Documentation: docs/dataloader/
• Optimizer Library Audits: docs/library_audit/
• DataLoader Standalone Library Audits: dataloader/docs/library_audit/
• DataLoader Architecture & User Guide: dataloader/README.md
• User Documentation: docs/en/

Authors & Maintenance

PROJECT LUMI (luminamoon) is engineered and maintained by:

• Silver Moon (@cloverxion)
• Lumina Moon (@cloverx-id)

License

Licensed under the Apache License, Version 2.0. See LICENSE for details.