apeiron-lm 0.0.3.post1

Byte-native parallel language model with unified latent reasoning

Apeiron LM

v0.0.3.1 — Byte-native parallel language model. Gradient-free SDPS training, Fourier global context, LoRA, multi-GPU, Flash Attention.

No tokenizer. No autoregressive loop. No vocabulary.

---

What is Apeiron LM?

Apeiron operates entirely at the byte level. Raw bytes (text, image, audio, video) are encoded into a continuous latent vector and decoded in a single parallel forward pass — no left-to-right loop, no tokenizer, no vocabulary.

Architecture

Input bytes  [0…255]
  → ByteEncoder          Transformer + mean-pool  →  v ∈ ℝᶜ
  → NoveltyEstimator     MLP gate                 →  novelty ∈ [0,1]
  → LatentExplorer       Prototype bank + proj    →  c* ∈ ℝᶜ
  → UnifiedLatentReasoner
        ├─ LengthPredictor    MLP → N_pred
        ├─ ContextBlock       (one of):
        │     none        CausalDWConv1D           O(N·k·C)
        │     replace     FourierSpectralConv1D    O(N·log(N)·C)
        │     parallel    Fourier + DWConv, gated  O(N·log(N)·C)
        │     adaptive    AdaptiveFourierFilter    O(N·log(N)·C) + O(C²)
        │     multiscale  MultiScaleFourierBlock   O(3N·log(N)·C)
        └─ ByteProjector  → N×256 byte logits
  → Output bytes

Hard invariants (never broken):

• Full parallelism — no sequential loop in forward or generation
• Byte-native — no tokenizer anywhere
• Output vocab = exactly 256
• Infinite effective context (Fourier modes)

---

Installation

pip install -e .

# Optional extras
pip install transformers          # model conversion from GPT-2, BERT, etc.
pip install Pillow                # image I/O
pip install flash-attn --no-build-isolation  # Flash Attention 2
pip install tensorboard wandb     # experiment tracking

---

Quick Start

from apeiron_lm import ApeironLM, ApeironConfig

model = ApeironLM(ApeironConfig.small())
print(model.generate_text("Hello", max_generate=64))

---

Training

Option A — SDPS (gradient-free, seconds)

from apeiron_lm import ApeironLM, ApeironConfig
from apeiron_lm.sdps import sdps_fit

model  = ApeironLM(ApeironConfig.small())
result = sdps_fit(model, data_tensor)   # (N, seq_len) LongTensor
print(result)
# SDPSResult(
#   elapsed_seconds=1.247,
#   n_samples=1024,
#   n_layers=4,
#   covariance_rank=128,
#   per_layer_residual_trace=['5.2134', '3.8901', '2.1244', '0.9912'],
#   per_head_reconstruction_error=
#     layer 0: ['0.142', '0.138', '0.151', '0.139']
#     ...
#   refine_steps_done=0,
# )

SDPS with gradient refinement (best of both worlds):

result = sdps_fit(
    model, data_tensor,
    refine_steps = 200,    # gradient steps on byte_proj + length_mlp only
    refine_lr    = 1e-4,   # encoder stays frozen
)

Stream from DataLoader:

result = sdps_fit(model, train_loader, max_batches=500)

Supervised output head (closed-form):

sdps   = SDPSTrainer(model)
sdps.fit(data_tensor)
W_out  = sdps.fit_output_head(X, y_labels, n_classes=10)

What SDPS does (v0.0.3.1 — all 8 quality gaps closed):

1. Captures encoder activations via forward hooks
2. Per-head covariance Σ_h for each attention head
3. Solves W_Q/K/V/O analytically per head (exact, not tiled)
4. FFN weights via target-propagation + ridge regression (not just eigenvectors)
5. Byte embedding seeded with sinusoidal encoding of byte values
6. byte_proj fitted via ridge regression (latent → byte histogram)
7. LatentExplorer prototypes seeded via k-means++ on training latents
8. length_mlp final layer fitted via ridge regression

Option B — Gradient training

from apeiron_lm import ApeironTrainer, TrainingArguments

args    = TrainingArguments(output_dir="./runs", num_epochs=10, batch_size=32)
trainer = ApeironTrainer(model, dataset, args)
trainer.train()

Option C — SDPS warm-start → gradient fine-tune

sdps_fit(model, data_tensor)      # analytical warm-start
trainer = ApeironTrainer(model, dataset, args)
trainer.train()                   # gradient refinement from SDPS weights

---

Fine-Tuning

Full fine-tuning

from apeiron_lm.finetune import finetune

result = finetune(
    model          = ApeironLM.load("./pretrained"),
    train_data     = dataset,
    output_dir     = "./finetuned",
    num_epochs     = 3,
    learning_rate  = 1e-5,
    layer_lr_decay = 0.9,      # encoder layers get progressively smaller LR
    freeze_encoder = False,
)

LoRA fine-tuning (parameter-efficient)

from apeiron_lm.finetune import ApeironFinetuner, FinetuneConfig

cfg = FinetuneConfig(
    output_dir          = "./lora_finetuned",
    num_epochs          = 3,
    learning_rate       = 2e-4,
    use_lora            = True,
    lora_r              = 8,
    lora_alpha          = 16.0,
    lora_dropout        = 0.05,
    lora_target_modules = ["in_proj", "out_proj", "linear1", "linear2"],
    lora_merge_after    = False,   # True = merge LoRA into weights after training
)
finetuner = ApeironFinetuner(model, dataset, cfg)
result    = finetuner.train()

LoRA manual API

from apeiron_lm.lora import (
    LoRAConfig, apply_lora, merge_lora, unmerge_lora,
    save_lora, load_lora, lora_parameter_count,
)

# Apply LoRA
lora_cfg = LoRAConfig(r=8, alpha=16, target_modules=["linear1", "linear2"])
apply_lora(model, lora_cfg)
trainable, total = lora_parameter_count(model)
print(f"Trainable: {trainable:,} / {total:,} ({100*trainable/total:.2f}%)")

# Train (only LoRA A, B matrices update)
trainer = ApeironTrainer(model, dataset, args)
trainer.train()

# Save only LoRA weights (~KB file)
save_lora(model, "./lora.pt")

# Load into a fresh model
model2 = ApeironLM.load("./pretrained")
apply_lora(model2, lora_cfg)
load_lora(model2, "./lora.pt")

# Merge for inference (no LoRA overhead)
merge_lora(model)

---

Fourier Global Context

from apeiron_lm import ApeironConfig, ApeironLM

cfg = ApeironConfig.base()

# Standard Fourier (infinite context, O(N log N C))
cfg.fourier_mode = "replace"

# Fourier + local DWConv in parallel (recommended)
cfg.fourier_mode = "parallel"

# Input-conditioned frequency filter (v0.0.3.1)
cfg.fourier_mode = "adaptive"

# Multi-resolution: coarse/mid/fine (v0.0.3.1, default for large)
cfg.fourier_mode = "multiscale"

# Channel-mixing (full C×C complex matrix per mode, v0.0.3.1)
cfg.fourier_mix_channels = True   # more expressive, use for small C

cfg.fourier_n_modes = 64   # number of frequency modes

model = ApeironLM(cfg)

Preset	fourier_mode	n_modes
tiny	none	16
small	none	32
base	parallel	64
large	multiscale	128

---

Attention Backends

from apeiron_lm.attention import (
    set_attention_backend,
    current_attention_backend,
    attention_diagnostics,
)

# Auto-select best available (Flash > SDPA > naive)
set_attention_backend("auto")

# Force a specific backend
set_attention_backend("flash")   # requires: pip install flash-attn
set_attention_backend("sdpa")    # PyTorch built-in
set_attention_backend("naive")   # always works

print(current_attention_backend())  # "naive" / "sdpa" / "flash"
print(attention_diagnostics())

# Or via config
cfg = ApeironConfig.base()
cfg.attention_backend = "sdpa"
model = ApeironLM(cfg)   # wires backend on init

---

Multi-GPU Training

DataParallel (single machine, simplest)

from apeiron_lm.parallel import wrap_data_parallel
from apeiron_lm import ApeironTrainer, TrainingArguments

model   = ApeironLM(cfg)
model   = wrap_data_parallel(model)   # wraps if >1 GPU
trainer = ApeironTrainer(model, dataset, args)
trainer.train()

DDP (recommended for multi-GPU)

torchrun --nproc_per_node=4 train.py

# train.py
from apeiron_lm.parallel import DDPContext, ApeironDDPTrainer

ctx     = DDPContext.detect()
model   = ApeironLM(cfg).to(ctx.device)
trainer = ApeironDDPTrainer(model, dataset, args, ctx=ctx)
trainer.train()
trainer.save("./output")   # rank 0 only

FSDP (large models)

from apeiron_lm.parallel import wrap_fsdp

model = ApeironLM(cfg)
model = wrap_fsdp(model, mixed_precision=True)

---

Model Conversion from Transformers

# Convert GPT-2 → Apeiron
apeiron-convert --from gpt2 --output ./my_model

# More options
apeiron-convert \
  --from bert-base-uncased \
  --output ./bert_model \
  --preset base \
  --steps 2000 \
  --verbose

from apeiron_lm.convert import ModelConverter, ConversionConfig

cfg    = ConversionConfig(
    teacher_name    = "gpt2",
    output_dir      = "./converted",
    preset          = "small",
    distill_steps   = 1000,
    teacher_layer   = -1,        # which teacher layer to distil (-1 = last)
    use_sdps_warmup = True,      # SDPS warm-start after distillation
)
result = ModelConverter(cfg).convert()
model  = ApeironLM.load(result["saved_to"])

Requires: pip install transformers

---

Multimodal

from apeiron_lm.multimodal import ApeironMultimodal, build_multimodal_input
from PIL import Image

mm = ApeironMultimodal(model)

# Text + image → text
result = mm.generate(
    text            = "Describe this image:",
    image           = Image.open("cat.png"),
    output_modality = "text",
    max_generate    = 128,
)
print(result["text"])

# Build raw multimodal byte tensor
inp = build_multimodal_input(
    text   = "Caption:",
    image  = Image.open("photo.jpg"),
)

Requires: pip install Pillow

---

Save / Load

model.save("./my_model")
model = ApeironLM.load("./my_model")

---

Configuration Reference

cfg = ApeironConfig(
    # Core
    latent_dim           = 128,
    encoder_layers       = 4,
    encoder_heads        = 4,
    encoder_ff_dim       = 512,
    n_max                = 512,

    # Context
    conv_kernel          = 5,
    fourier_mode         = "parallel",   # none/replace/parallel/adaptive/multiscale
    fourier_n_modes      = 64,
    fourier_mix_channels = False,        # True = full C×C complex per mode

    # Attention
    attention_backend    = "auto",       # auto/flash/sdpa/naive

    # Explorer
    n_prototypes         = 64,
    explorer_threshold   = 0.5,
    explorer_warmup_steps = 500,

    # Training
    len_hidden           = 64,
    dropout              = 0.1,
)
cfg.validate()

---

Model Presets

Preset	~Params	latent_dim	enc_layers	fourier_mode
tiny	~1M	64	2	none
small	~6M	128	4	none
base	~25M	256	6	parallel
large	~100M	512	8	multiscale

---

Known Limitations

Causal Fourier: FourierSpectralConv1D uses full (non-causal) FFT. The model is designed for parallel decoding so strict causality is not required during training. If you need strict AR inference, use fourier_mode="none" (CausalDWConv1D is truly causal).

Flash Attention integration: Flash Attention is available as a backend for efficient_attention() but is not yet wired directly into the ByteEncoder's MultiheadAttention module (PyTorch MHA does not expose a custom attention kernel interface). The set_attention_backend() API is fully functional for custom attention calls; full MHA integration is planned for v0.0.4.

SDPS sparse data: SDPS requires n_samples >> latent_dim for a well-conditioned covariance. Minimum recommended: n_samples ≥ 4 × latent_dim.

Multi-GPU / activation capture: SDPSTrainer._capture_activations() does not yet support DDP (model must be on a single device during SDPS). Run SDPS before wrapping with DDP.

Video decoding: bytes_to_video() is not yet implemented. Only video_frames_to_bytes() (encode) is available.

FSDP + LoRA: FSDP wrapping after apply_lora() may encounter issues with parameter flattening. Recommended order: apply LoRA → merge weights → wrap FSDP. Or use DDP instead.

---

Version History

Version	Highlights
v0.0.3.1	SDPS 8-gap fix, Fourier channel-mix/adaptive/multiscale, LoRA, multi-GPU, Flash Attention, 299 tests
v0.0.3	SDPS gradient-free training, Fourier global context, model conversion, fine-tuning API, multimodal
v0.0.2	CausalDWConv1D, LatentExplorer, HuggingFace integration, 78 tests
v0.0.1	Initial byte-native architecture, ByteEncoder, parallel decoding

---

License

Apache 2.0 — see LICENSE.

Author

Ömür Bera Işık, 2026.