🚀 On-The-Fly Weight Synthesizer (OTF-LLM Engine)

High-performance hybrid LLM inference engine featuring custom Fused OpenAI Triton INT4 GEMM kernels, Outlier-Aware weight quantization, Zero-RAM Incremental Quantizer (<150MB RAM), 98.16% Scientific Logit Parity, 3-Tier Hierarchical MoE Offloader [EXPERIMENTAL], Companion Long-Term Memory, and an Interactive Gradio Web UI.

🔬 Engineering & Research: GT Labs AI

Developed and maintained by GT Labs AI.

• 🚀 GT Labs AI Mission: Ultra-fast MVP engineering, custom AI integration, and deep neural network optimization research.
• 👨‍💻 Author & Lead AI Engineer: Gleb Tikhiy (@GlebTikhiy)
• 📧 Contact & Inquiries: team.gtlabs@gmail.com
• 🌐 Organization: GT Labs AI on GitHub

🎯 Project Goal & Key Breakthroughs

Overcoming memory-bound bottlenecks and hardware VRAM constraints when executing Large Language Models (LLMs) on consumer GPUs (RTX 3060 / 4060 / 5060 Ti).

Instead of transferring heavy FP16 weights from VRAM, OTF-LLM Engine performs hardware-accelerated dequantization of Outlier-Aware INT4 weights directly inside GPU registers (SRAM) via custom OpenAI Triton GEMM Kernels, streams weights incrementally without RAM allocation (< 150 MB RAM peak), and integrates long-term user memory (companion_memory.py).

🔬 Formal Scientific Logit Parity Benchmark (20 Prompts)

Evaluated via tests/test_formal_parity.py comparing Base FP16 vs OTF INT4 Champion Engine across 20 multi-domain prompts:

• 📐 Average Logit Cosine Similarity: 98.1556%
• 🎯 Top-1 Exact Token Match Rate: 70.0% (14/20 exact token identity)
• 📉 Average KL-Divergence: 0.2154

📊 Performance Benchmark (NVIDIA RTX 5060 Ti 16GB)

Benchmarking conducted on an NVIDIA GeForce RTX 5060 Ti 16GB GPU:

🏛️ Architecture & Key Innovations

[Input Vector X] ──► [Global Static Permutation (global_perm_idx)]
                                   │
         ┌─────────────────────────┴─────────────────────────┐
         ▼                                                   ▼
[Outlier Channels (1% FP16)]                 [Background Block (99% INT4)]
   │                                                         │
   ├──► Pure FP16                                            ├──► Range [-7 ... +7]
   └──► Input X_outliers                                     ├──► 2:1 Packing (uint8)
                                                             └──► Zero-Point = 0 BYTES!
                                                                     │
                                                                     ▼
                                                      [Custom Fused Triton GEMM Kernel]
                                                      (Dequantization in GPU SRAM Registers)
                                                                     │
         ┌───────────────────────────────────────────────────────────┘
         ▼
[Continuous GEMM Addition: Outliers + Triton Background = Exact FP16 Output]

1. Custom Outlier-Aware Fused Triton GEMM Kernel (otf_triton_kernel.py): Packed uint8 weights are streamed from VRAM and dequantized directly inside GPU chip registers (SRAM) during matrix multiplication, eliminating temporary FP16 tensor allocations in VRAM.
2. Incremental Sharded Quantizer with Resume Support (convert_global_universal.py): Memory-mapped (mmap) sharded quantizer with < 150 MB CPU RAM footprint. Saves intermediate disk chunks to enable crash-resilient resuming.
3. Global Static Permutation (global_perm_idx) & Outlier Preservation: A unified channel permutation table across the model (1.6 MB VRAM). Isolating the Top-1% critical outlier channels ($|W| \times |X_{\text{profile}}|$) in FP16 completely suppresses quantization noise and guarantees 98.16% logit similarity.
4. Companion Long-Term Memory Manager (companion_memory.py): Zero-VRAM, lightweight CPU RAM module that automatically extracts and retrieves user facts in < 2 ms via TF-IDF cosine similarity, injecting relevant facts into system prompts.
5. 3-Tier Hierarchical MoE Offloader [EXPERIMENTAL] (otf_moe_offloader.py): R&D module for executing Mixture-of-Experts models via a 3-tier memory hierarchy (Disk ➔ CPU RAM ➔ GPU VRAM).
6. Gradio Web Demo (otf-demo) & FastAPI REST API (otf-server): Includes a built-in interactive browser Web UI with real-time VRAM allocation counters and OpenAI API spec compatibility (/v1/chat/completions).

📁 Repository Structure

otf-llm-engine/
├── setup.py                                  # Setuptools package configuration
├── pyproject.toml                            # PEP 517/518 build system
├── MANIFEST.in                               # Package assets configuration
├── pipeline_run.py                           # Automated 1-click end-to-end pipeline
├── otf_llm/                                  # Main python package namespace
│   ├── __init__.py                           # Module entry point (v3.2.0)
│   ├── make_profile_universal.py             # Activation profile calibrator
│   ├── convert_global_universal.py           # Incremental mmap safetensors quantizer
│   ├── run_triton_universal.py               # Triton GEMM inference runner
│   ├── otf_triton_kernel.py                  # Custom Fused Triton INT4 GEMM kernel
│   ├── companion_memory.py                   # Zero-VRAM long-term user memory store
│   ├── web_demo.py                           # Interactive Gradio Web UI
│   ├── otf_moe_offloader.py                  # [EXPERIMENTAL] 3-Tier MoE Offloader
│   ├── direct_quantized_importer.py          # [EXPERIMENTAL] AWQ/GPTQ importer
│   ├── query_guided_sparse_kv.py             # Context retrieval (CPU RAM -> GPU)
│   ├── otf_context_compressor.py             # SnapKV / KIVI cache compressor
│   └── server_fastapi.py                     # REST API server (OpenAI API + SSE)
├── tests/                                    # Benchmark & validation test suites
│   ├── test_formal_parity.py                 # Scientific logit parity benchmark (20 prompts)
│   ├── validate_llama3_2_3b.py               # Llama-3.2-3B validation runner
│   ├── test_intelligence_suite.py            # Intelligence and logic reasoning suite
│   ├── test_base_model_suite.py              # Baseline FP16 model suite
│   └── test_client.py                        # SSE streaming test client
├── README.md                                 # Project documentation
└── LICENSE                                   # MIT License

🛠️ Quickstart & Installation

Official PyPI Installation

pip install otf-llm

Launch Interactive Web Demo

otf-demo

Run Formal Scientific Parity Test

python tests/test_formal_parity.py

📜 License

Distributed under the MIT License. See LICENSE for details.