glyd-gpu

The GPU half of Glyd: a bf16 model's weights held compressed in GPU memory, bit for bit, and multiplied from there (about a third less GPU memory), for PyTorch and transformers, and as a vLLM quantization method.

pip install "glyd[gpu]"      # glyd.from_pretrained, glyd.fit
pip install "glyd[vllm]"     # vllm serve MODEL --quantization glyd; glyd run MODEL

This package is what those two extras install: glyd[gpu] is glyd-gpu[gpu] (PyTorch, transformers, accelerate) and glyd[vllm] is glyd-gpu[vllm] (vLLM 0.30); installed on its own it needs none of them. It ships as compiled wheels for Linux with an NVIDIA GPU (Ampere or later; CUDA 12 or 13 builds of PyTorch). Import name: glyd_gpu; python -m glyd_gpu has fit, pack and verify.

C, C++, Rust

The wheel carries the GPU libraries (libglyd_gpu_cuda12.so and libglyd_gpu_cuda13.so: they need only the NVIDIA driver, and are built with the CUDA 12 and 13 runtimes linked in) and their C header, glyd_gpu.h, in the package's directory:

python -c "import glyd_gpu, os; print(os.path.dirname(glyd_gpu.__file__))"

For a program that multiplies by the matrices of a model saved by glyd pack (or glyd.save_pretrained), or decodes them back to bf16 bit for bit, without Python: the functions the header declares take the arrays of a pack as the save names them, passed as they are, and choose how to run each product themselves. Linking: gcc prog.c -I DIR -I CUDA/include -L DIR -lglyd_gpu_cuda13 -L CUDA/lib64 -lcudart -Wl,-rpath,DIR.

License: Business Source License 1.1 (LICENSE). Its additional use grant covers personal, educational, research and other non-commercial production use; commercial production use needs a separate license from the licensor.