NeuralOps

NeuralOps is a production-grade, mathematically verified framework for linear attention mechanisms, model compression engines, and fused Triton kernels.

📑 Table of Contents

1. Key Highlights
2. Installation
3. Quick Start
4. Architecture & Data Flow
5. Attention Mechanism Zoo (159)
6. Compression Engine Registry (73)
7. Fused Triton Kernels
8. Supported Architectures
9. Performance & Tradeoffs
10. API Reference
11. Benchmarking CLI
12. FAQ
13. Contributing
14. Citation & License

Key Highlights

• 159 Attention Mechanisms: Canonical linear, sparse, gated, hybrid, SSM, and SVD factorized attention layers.
• 73 Model Compressors: SVD truncation, activation-calibrated decomposition (ASVD/AdaSVD), structured pruning, and quantization.
• 21 Fused Triton Kernels: Highly optimized GPU execution with seamless, transparent PyTorch CPU/MPS fallbacks.
• Universal Architecture Surgery: Drop-in zero-shot attention replacement across popular Transformer models.
• Extensive Backwards Compatibility: 657 preserved deprecated aliases for seamless backward compatibility.

Installation

Requirements

Platform Matrix

Installation via PyPI

# Standard install
pip install neuralops

# Install from source
git clone https://github.com/tahamajs/SVD_linear_Attention.git
cd SVD_linear_Attention
pip install -e ".[dev]"

Verify your installation:

python -c "import neuralops; print(neuralops.__version__)"
python -m neuralops.benchmarks verify

Quick Start

Recipe 1: 3-Line Attention Surgery

Replace quadratic attention in HuggingFace models with subquadratic SVD Linear Attention:

import torch
from transformers import AutoModelForSequenceClassification
from neuralops.utils.replace_attention import replace_attention

model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased")
model = replace_attention(model, "svd_linear_attention", svd_rank=128)

inputs = torch.randint(0, 1000, (2, 64))
outputs = model(inputs)
print("Logits shape:", outputs.logits.shape)

Architecture Overview

graph TD
    A[Pretrained Model] --> B[replace_attention / surgery]
    B --> C{Architecture Adapter}
    C -->|DistilBERT| D[Flat MultiHead Attention]
    C -->|BERT/RoBERTa| E[Nested Self-Attention]
    C -->|LLaMA/Qwen| F[GQA / RoPE Projections]
    D --> G[NeuralOps BaseAttention Implementation]
    E --> G
    F --> G
    G --> H[Execution Engine: Triton or PyTorch Fallback]

Attention Mechanism Zoo (159)

Compression Engine Registry (73)

Fused Triton Kernels

NeuralOps implements 21 custom Triton kernels with automatic fallback:

• Sliding Window Attention (sliding_window.py)
• Paged Attention & KV Cache (paged_attention.py)
• Flash-Decoding Split-KV (linear_attn.py)
• Fused RoPE & Rotary Embeddings (varlen_attention.py)
• State Space & Selective Scan (selective_scan.py, state_update.py)

All kernels automatically fall back to native PyTorch ops when running on non-CUDA hardware or CPU.

Supported Architectures

Performance and Tradeoffs

See docs/REPRODUCIBILITY.md for benchmark setup and reproduction details.

API Reference

from neuralops.core.registry import ATTENTION_REGISTRY, COMPRESSOR_REGISTRY

# Registry lookups
attention_cls = ATTENTION_REGISTRY.get("cosformer")
compressor_cls = COMPRESSOR_REGISTRY.get("slicegpt")

# Check registered counts
print(len(ATTENTION_REGISTRY))   # 159
print(len(COMPRESSOR_REGISTRY))  # 73

Detailed guides:

• API Reference
• Architecture Details
• Troubleshooting
• Migration Guide

Benchmarking CLI

# Verify system & hardware setup
python -m neuralops.benchmarks verify

# List all supported attention & compressor modules
python -m neuralops.benchmarks list

# Run a benchmark sweep
python -m neuralops.benchmarks run --module svd_linear_attention --model distilbert-base-uncased --dataset sst2

FAQ

Q: Does NeuralOps require an NVIDIA GPU?
A: No. PyTorch CPU fallbacks ensure 100% of the attention mechanisms and compression engines run seamlessly on macOS (including Apple Silicon MPS) and CPU-only environments.

Q: Are deprecated module names supported?
A: Yes. All 657 legacy aliases map automatically to their canonical implementations and emit informative deprecation warnings.

Contributing

Contributions are welcome! Please review CONTRIBUTING.md and CODE_OF_CONDUCT.md before submitting pull requests.

Citation and License

@software{neuralops2026,
  title  = {NeuralOps: SVD Linear Attention & Model Compression Framework},
  author = {Majs, Taha},
  year   = {2026},
  url    = {https://github.com/tahamajs/SVD_linear_Attention}
}

Distributed under the MIT License. See LICENSE for details.