sykra 0.1.0

2. README.md

# Sykra

[![PyPI Version](https://img.shields.io/pypi/v/sykra.svg)](https://pypi.org/project/sykra/)
[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
[![Python Version](https://img.shields.io/badge/Python-3.8%2B-green.svg)](https://www.python.org/)
[![PyTorch](https://img.shields.io/badge/PyTorch-2.0%2B-ee4c2c.svg)](https://pytorch.org/)
[![Hardware](https://img.shields.io/badge/Hardware-NVIDIA_CUDA_%7C_Google_TPU-7b1fa2.svg)](#hardware-acceleration)

**Sykra** is a high-performance, sub-quadratic hybrid **Decoder-Only Transformer** architecture designed for efficient long-context language modeling. 

It combines **Gated DeltaNet** (chunk-parallel linear associative memory) with **Standard Multi-Head Attention** in a periodic **3:1 layer ratio** (3 Delta layers followed by 1 Full Attention layer).

---

## Architectural Highlights

- **3:1 Periodic Hybridization:** 75% of layers operate with $O(N)$ linear recurrence, and 25% operate with global FlashAttention / SDPA for long-range associative recall.
- **Chunkwise Parallel Training:** Utilizes the **WY / Unit Triangular Transform** formulation to parallelize inner-chunk recurrence into GEMM matrix operations via `torch.linalg.solve_triangular` (GPU) and XLA-native matrix inversion (TPU).
- **Data-Dependent Forget Gating ($\alpha_t$):** Dynamically scales state retention to bound memory spectral radius and prevent numerical explosion.
- **$O(1)$ Inference Memory Footprint:** Delta layers update a fixed-size $d \times d$ associative matrix during generation, eliminating unbounded KV cache growth.
- **Modern LLM Standards:** Grouped-Query Attention (GQA), Rotary Position Embeddings (RoPE), RMSNorm, and SwiGLU MLP blocks.
- **100% Hugging Face Native:** Fully compatible with `PreTrainedModel`, `GenerationMixin`, `safetensors`, and `AutoModelForCausalLM`.

---

## Computational Complexity

| Layer Type | Training Time | Inference Step Time | Inference Memory | Context Scaling |
| :--- | :--- | :--- | :--- | :--- |
| **Standard Attention** | $O(N^2 \cdot d)$ | $O(N \cdot d)$ | $O(N \cdot d)$ (KV Cache) | Quadratic |
| **Sykra Delta Layer** | $O(N \cdot d)$ | $O(d^2)$ | $O(d^2)$ (Fixed State) | **Linear** |
| **Sykra Hybrid (3:1)** | **$\approx 0.25 \cdot O(N^2) + 0.75 \cdot O(N)$** | **Sub-quadratic** | **75% KV Cache Reduction** | **Sub-quadratic** |

---

## Installation

### Standard Installation (CUDA / CPU)
```bash
pip install sykra

Google Cloud TPU Installation

pip install sykra[tpu]

Build from Source

git clone https://github.com/sykra-ai/sykra.git
cd sykra
pip install -e .

Quick Start

1. Basic Generation with Hugging Face AutoModel

import torch
from transformers import AutoTokenizer, AutoConfig, AutoModelForCausalLM
from sykra import SykraConfig, SykraForCausalLM

# Register Sykra with Hugging Face
AutoConfig.register("sykra", SykraConfig)
AutoModelForCausalLM.register(SykraConfig, SykraForCausalLM)

tokenizer = AutoTokenizer.from_pretrained("gpt2")
config = SykraConfig(
    vocab_size=tokenizer.vocab_size,
    hidden_size=768,
    num_hidden_layers=12,      # 9 Delta Layers + 3 Full Attention Layers
    num_attention_heads=12,
    num_key_value_heads=4,     # Grouped-Query Attention (3:1)
    chunk_size=64,
    delta_ratio=3
)

model = SykraForCausalLM(config).cuda()

# Autoregressive generation
prompt = "Sykra architecture is designed to"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

with torch.no_grad():
    generated_ids = model.generate(
        **inputs,
        max_new_tokens=40,
        use_cache=True,
        do_sample=True,
        temperature=0.7,
        pad_token_id=tokenizer.eos_token_id
    )

print(tokenizer.decode(generated_ids[0], skip_special_tokens=True))

2. Training with Mixed Precision (GPU)

You can train custom Sykra models using standard PyTorch loops or Hugging Face Trainer:

from sykra import SykraConfig, SykraForCausalLM
import torch

config = SykraConfig(
    vocab_size=50257,
    hidden_size=768,
    intermediate_size=2048,
    num_hidden_layers=12,
    chunk_size=64,
    delta_ratio=3
)

model = SykraForCausalLM(config).cuda()
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4)

# Synthetic batch: (Batch Size: 4, Sequence Length: 512)
input_ids = torch.randint(0, config.vocab_size, (4, 512)).cuda()
labels = input_ids.clone()

# Forward & Backward pass with AMP
with torch.cuda.amp.autocast(dtype=torch.bfloat16):
    outputs = model(input_ids=input_ids, labels=labels)
    loss = outputs.loss

loss.backward()
optimizer.step()
print(f"Training Step Loss: {loss.item():.4f}")

3. Distributed TPU Training (Google Cloud TPU v4 / v5e / v6e)

Sykra includes a native PyTorch/XLA distributed engine that prevents dynamic graph recompilations:

from transformers import AutoTokenizer
from sykra import SykraConfig, train_on_tpu

tokenizer = AutoTokenizer.from_pretrained("gpt2")
config = SykraConfig(
    vocab_size=tokenizer.vocab_size,
    hidden_size=768,
    num_hidden_layers=12,
    chunk_size=64,
    delta_ratio=3
)

# Automatically scales across available TPU cores (v5e-1, v5e-8, TPU Pods)
train_on_tpu(
    config=config,
    dataset_path="dataset.txt",
    tokenizer=tokenizer,
    output_dir="./sykra_tpu_checkpoint",
    batch_size_per_core=8,
    epochs=5,
    learning_rate=4e-4
)

Model Architecture Specification

SykraModel(
  (embed_tokens): Embedding(vocab_size, hidden_size)
  (layers): ModuleList(
    (0-2): 3 x SykraDecoderLayer [Gated DeltaNet Layer] -> Linear Recurrence
    (3):   1 x SykraDecoderLayer [Full Attention Layer] -> FlashAttention / RoPE / GQA
    (4-6): 3 x SykraDecoderLayer [Gated DeltaNet Layer] -> Linear Recurrence
    (7):   1 x SykraDecoderLayer [Full Attention Layer] -> FlashAttention / RoPE / GQA
    ...
  )
  (norm): SykraRMSNorm()
  (lm_head): Linear(hidden_size, vocab_size)
)

License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.

Citation

If you use Sykra in your research or production workloads, please cite:

@software{sykra2026,
  author = {Sykra AI Team},
  title = {Sykra: A 3:1 Hybrid Gated DeltaNet and Attention Transformer Architecture},
  url = {https://github.com/sykra-ai/sykra},
  version = {0.1.0},
  year = {2026}
}