quicksilver-inference 0.1.0

High-performance GGUF model inference with quantized kernels

Quicksilver

High-Performance GGUF Inference Engine

Quicksilver is a standalone, production-ready library for GGUF quantized model inference. It achieves 84 tok/s on CPU (95% faster than llama.cpp) through optimized C++ kernels with AVX2 SIMD and OpenMP parallelization.

Performance

Model	Size	Quicksilver	llama.cpp	Speedup
SmolLM2-135M	101MB	84 tok/s	43 tok/s	+95%
Qwen2.5-1.5B	1.0GB	12 tok/s	~12 tok/s	comparable

Benchmarked with OMP_NUM_THREADS=8 on Intel CPU

Backends

Backend	Platform	Status
CPU	All (AVX2/NEON)	✅ Production
CUDA	NVIDIA GPUs	✅ Ready
Metal	Apple Silicon	✅ Ready
CANN	Huawei Ascend NPU	✅ Ready

Features

• Native GGUF Parsing - Zero external dependencies for model loading
• 9 Quantization Types - Q4_0, Q5_0, Q8_0, Q2_K, Q3_K, Q4_K, Q5_K, Q6_K, F16
• AVX2 SIMD - Vectorized Q4_K/Q6_K kernels with FMA intrinsics
• OpenMP Parallelization - Multi-threaded GEMV operations
• Fused Operations - Minimal Python overhead with C++ transformer forward pass
• Auto Backend Selection - Automatically uses best available (CUDA > Metal > CPU)

Installation

CPU (All Platforms)

cd quicksilver
pip install -e .

# Build C++ kernels (requires C++ compiler + OpenMP)
cd csrc && python setup_quantized.py build_ext --inplace

# Optimal performance
export OMP_NUM_THREADS=8

CUDA (NVIDIA GPUs)

# Requires: CUDA toolkit, nvcc compiler
cd quicksilver/csrc
python setup_cuda.py build_ext --inplace

Metal (macOS)

# Requires: Xcode (not just Command Line Tools)
python quicksilver/backends/metal/compile_shaders.py

CANN (Huawei Ascend NPU)

# Requires: CANN toolkit from https://www.hiascend.com/cann
# Install torch_npu first
pip install torch-npu

# Build CANN kernels
cd quicksilver/backends/cann
python setup_cann.py build_ext --inplace

Quick Start

Python API

from quicksilver import QuantizedEngine

# Load model
engine = QuantizedEngine("model.gguf")

# Generate text
tokens = engine.generate("Hello, world!", max_tokens=50)
print(engine.decode(tokens))

CLI

# Generate text
quicksilver generate -m model.gguf -p "Once upon a time"

# Run benchmark
quicksilver benchmark -m model.gguf -n 100

# Show model info
quicksilver info -m model.gguf

Architecture

quicksilver/
├── core/           # GGUF parsing and quantization
│   ├── parser.py   # Native GGUF file parser
│   ├── tensor.py   # Quantized tensor operations
│   └── quantization.py  # Quantization type definitions
├── csrc/           # C++ kernels
│   └── fused_quantized.cpp  # Optimized transformer kernels
├── quantized_engine.py  # Main inference engine
└── cli.py          # Command-line interface

Supported Quantization Types

Type	Bits	Block Size	Status
Q4_0	4.0	32	✅ Native GEMV
Q5_0	5.0	32	✅ Native GEMV
Q8_0	8.0	32	✅ Native GEMV
Q2_K	2.5	256	✅ Native GEMV
Q3_K	3.4	256	✅ Native GEMV
Q4_K	4.5	256	✅ AVX2 SIMD
Q5_K	5.5	256	✅ Native GEMV
Q6_K	6.5	256	✅ AVX2 SIMD
F16	16	1	✅ Native GEMV

Roadmap

• CPU inference with quantized kernels
• Beat llama.cpp performance (84 vs 43 tok/s = +95%)
• Support 9 quantization types (covers 95%+ of GGUF models)
• Metal GPU backend shaders
• CUDA GPU backend kernels
• OpenAI-compatible API server
• Streaming generation
• Batch inference

Development

# Install dev dependencies
pip install -e ".[dev]"

# Run tests
pytest tests/

# Format code
black quicksilver/
ruff check quicksilver/

License

Apache 2.0