qorva 0.1.0

Hybrid Mamba-2 + Attention + MoE language models, easy to load and run.

qorva

Hybrid Mamba-2 + Attention + Mixture-of-Experts language models — easy to load, run, and fine-tune.

pip install qorva

Quick start

from qorva import load_pretrained, generate

model = load_pretrained("your_username/qorva-model-fast")
print(generate(model, "Once upon a time in a small village,", max_new_tokens=100))

Or, using the bound method directly:

from qorva import load_pretrained

model = load_pretrained("your_username/qorva-model-fast")
print(model.generate("Once upon a time", max_new_tokens=100, temperature=0.8))

Command line

qorva-generate --model your_username/qorva-model-fast --prompt "Once upon a time"

Build from scratch

from qorva import QorvaModel, QorvaConfig

cfg   = QorvaConfig.nano_full()      # ~370M, Mamba + Attention + MoE
model = QorvaModel(cfg)
print(f"{model.num_parameters()/1e6:.1f}M parameters")

Available presets:

• QorvaConfig.micro() — ~32M, for fast experimentation
• QorvaConfig.nano_full() — ~370M, Mamba + Attention + MoE
• QorvaConfig.nano_baseline() — ~370M, Attention-only baseline

Push your trained model to HuggingFace

from qorva import push_to_hub

push_to_hub(model, "your_username/qorva-nano-370m", token="hf_...")

Perplexity evaluation

from qorva.utils.generation import compute_perplexity

ppl = compute_perplexity(model, "Some evaluation text here...")
print(f"PPL: {ppl:.2f}")

Architecture

Each layer mixes two branches with learnable weights:

x' = x + alpha * Mamba2(LN(x)) + beta * GQAttention(LN(x))
x  = x' + LatentMoE(LN(x'))      # stage="full" only

Stage	Branches
attn_only	Attention + FFN
mamba_only	Mamba + FFN
hybrid	Mamba + Attention + FFN
full	Mamba + Attention + LatentMoE

License

Apache-2.0