6 projects
blt-trainer
Pure functional 96-layer Byte Latent Transformer trainer with Muon, DiLoCo HF Buckets, Open-SLM benchmarks, and Axiomic curriculum
slm-parallax
Benchmarking MaxText vs PyTorch on NVIDIA Blackwell (RTX PRO 6000 96GB) across BF16, FP8, and NVFP4
fiveattn
100M-parameter Transformer comparing 5-Component Attention (GAU) and MHA, with SmolLM tokenizer, Cut-Cross-Entropy, SwiGLU, and RMSNorm
slm388-torchdiff
Masked Diffusion Language Model (MDLM) pretraining in PyTorch with FSDP2, Muon optimizer, and Hugging Face Jobs support.
cupydiff
High-Performance Discrete Diffusion Language Model in Triton + CuPy (Functional, No PyTorch)
adjaxt
Distributed training with JAX on free compute, for humans
