slm-packager 0.2.1

A unified runtime and developer layer for Small Language Models

SLM Packager

A Unified Runtime & Developer Layer for Small Language Models

SLM Packager is an open-source toolkit for running, packaging, and evaluating Small Language Models (1B-7B parameters) across different formats and runtimes. Think of it as Terraform for SLMs — making model deployment simple, reproducible, and developer-friendly.

✨ Features

• 🎯 Model Registry: One-command downloads from HuggingFace with slm pull
• 🔄 Multi-Runtime Support: llama.cpp (GGUF), Transformers (PyTorch), ONNX
• ⚡ GPU Acceleration: MPS (Apple Silicon), CUDA (NVIDIA), Metal (llama.cpp)
• ⚙️ Auto-Quantization: On-device model quantization with automatic tool setup
• 📊 Benchmarking: Measure speed, memory, latency across runtimes
• 🛠️ Config-Driven: YAML configs for reproducible deployments
• 🌐 API Server: FastAPI-based serving with streaming support

🚀 Quick Start

Installation

git clone https://github.com/Ayo-Cyber/slm-packager.git
cd slm-packager
pip install -e .

Pull & Run a Model

# List available models
slm list

# Pull GPT-2 (500MB, fast for testing)
slm pull gpt2

# Run it
slm run gpt2 --prompt "Explain AI in one sentence"

That's it! The model downloads, auto-configures, and runs.

Pull a GGUF Model

# Pull TinyLlama with llama.cpp (637MB)
slm pull tinyllama

# Run with different parameters
slm run tinyllama --prompt "Write a haiku"

📦 Available Models

Model	Size	Runtime	Description
gpt2	500MB	transformers	OpenAI GPT-2, fast to download
tinyllama	637MB	llama.cpp	1.1B chat model, CPU-optimized
phi-2	1.6GB	llama.cpp	Microsoft's 2.7B reasoning model
qwen-1.8b	1.1GB	llama.cpp	Alibaba's efficient chat model

View all: slm list
Pull with specific quantization: slm pull tinyllama --quant q8_0

🛠️ CLI Commands

# Model management
slm list                    # Show available models
slm list --installed        # Show downloaded models
slm pull <model>            # Download a model
slm pull <model> --list-variants  # Show quantization options

# Running models
slm run <model> --prompt "Your prompt"
slm run <config.yaml> --prompt "Your prompt"

# Quantization (auto-downloads tool)
slm quantize input.gguf output.gguf --type q4_k_m

# Benchmarking
slm benchmark <model>

# API server
slm serve --port 8000

# Manual config creation
slm init

⚡ GPU Acceleration

SLM Packager supports GPU acceleration across different hardware platforms.

Apple Silicon (MPS) - Zero Setup Required! 🍎

No installation needed - works out of the box on M1/M2/M3 Macs!

# Create GPU-accelerated config
slm init --name gpt2 --path gpt2 --format transformers --runtime transformers --device mps -o gpt2-gpu.yaml

# Run on GPU
slm run gpt2-gpu.yaml --prompt "Explain quantum computing"

Real Performance (M2 Pro):

GPT-2 Performance Comparison
├─ CPU:  1.3 tokens/sec
└─ MPS:  2.4 tokens/sec  ⚡ 2.14x faster!

Tested on: M2 Pro, macOS 14.x, GPT-2 (124M parameters)

Requirements:

• macOS 12.3 or later
• Apple Silicon (M1/M2/M3 series)
• PyTorch 1.12+ (included with installation)

NVIDIA GPU (CUDA)

# Install llama.cpp with CUDA support
CMAKE_ARGS="-DLLAMA_CUBLAS=on" pip install llama-cpp-python --no-cache-dir

# Set gpu_layers in config
runtime:
  type: llama_cpp
  device: cuda
  gpu_layers: 32  # Offload layers to GPU

Expected Performance:

• 2-5x speedup vs CPU
• Depends on GPU, model size, and layers offloaded

llama.cpp Metal (Apple Silicon)

For GGUF models with llama.cpp on Apple Silicon:

# Rebuild with Metal support
CMAKE_ARGS="-DLLAMA_METAL=on" pip install llama-cpp-python --no-cache-dir

# Use GPU layers in config
runtime:
  type: llama_cpp
  device: cpu  # Metal auto-detected
  gpu_layers: 32  # Offload to GPU

---

📊 Performance Benchmarks

Real-world performance on different hardware:

GPT-2 (124M parameters)

Runtime	Device	Tokens/sec	Memory	Notes
transformers	CPU (M2 Pro)	1.3	2.1GB	Baseline
transformers	MPS (M2 Pro)	2.4	2.1GB	2.14x speedup ⚡
ONNX	CPU (M2 Pro)	13.8	600MB	With KV-cache
llama.cpp	CPU	~15-20	~400MB	Quantized GGUF

TinyLlama (1.1B parameters)

Runtime	Device	Tokens/sec	Memory	Notes
llama.cpp	CPU	15-20	~800MB	Q4_K_M quantization
llama.cpp	Metal (M1)	40-60	~800MB	With GPU layers
transformers	CPU	5-10	4GB	Full precision

Performance varies based on hardware, model size, and configuration. Benchmarks collected on M2 Pro (Dec 2025).

---

📖 Runtime Comparison

Choose the right runtime for your use case:

Runtime	Best For	Pros	Cons
llama.cpp	Production, efficiency	Fast, low memory, quantized	GGUF format only
transformers	Development, flexibility	Latest models, GPU support	Higher memory
ONNX	Cross-platform, optimization	Fast, portable, optimized	Requires model export

When to Use Each

llama.cpp (GGUF):

• ✅ Production deployments
• ✅ Limited memory/CPU
• ✅ Want quantization
• ✅ Edge devices

transformers (PyTorch):

• ✅ Development & experimentation
• ✅ Latest HuggingFace models
• ✅ Fine-tuning workflows
• ✅ GPU available

ONNX:

• ✅ Cross-platform deployment
• ✅ ML pipeline integration
• ✅ Optimized inference graphs
• ✅ Already have ONNX models

---

🎯 Example Workflows

Developer: Fine-Tune & Quantize

# 1. Fine-tune your model (external tool)
# 2. Quantize it
slm quantize my-model.gguf my-model-q4.gguf --type q4_k_m

# 3. Test it
slm run my-model-q4.gguf --prompt "Test prompt"

# 4. Benchmark it
slm benchmark my-model-q4.gguf

Researcher: Compare Runtimes

# Pull same model, different runtimes
slm pull gpt2              # Transformers
slm pull tinyllama         # llama.cpp

# Benchmark both
slm benchmark gpt2
slm benchmark tinyllama

# Compare results

MacBook User: GPU-Accelerated Inference

# Zero setup - just run!
slm pull gpt2
slm init --name gpt2 --path gpt2 --device mps -o gpt2-gpu.yaml
slm run gpt2-gpu.yaml --prompt "Hello!"

# 2.14x faster than CPU! ⚡

ONNX User: Export & Run

# 1. Export model to ONNX
pip install "optimum[exporters]"
optimum-cli export onnx --model gpt2 --task text-generation-with-past models/gpt2-onnx/

# 2. Create config  
slm init --name gpt2 --path models/gpt2-onnx --format onnx --runtime onnx -o gpt2-onnx.yaml

# 3. Run (13.8 tok/s on CPU!)
slm run gpt2-onnx.yaml --prompt "Hello world"

---

📦 Configuration

Example Config

# my-model.yaml
model:
  name: my-custom-model
  path: /path/to/model.gguf
  format: gguf
  description: "My quantized model"

runtime:
  type: llama_cpp
  device: cpu
  threads: 8
  context_size: 2048
  gpu_layers: 0

params:
  temperature: 0.7
  top_p: 0.9
  top_k: 40
  max_tokens: 512
  stream: true
  stop: ["User:", "\n\n"]

Create Config Interactively

slm init  # Guided prompts

---

🌐 API Server

Start a FastAPI server for HTTP access:

# Start server
slm serve --port 8000

# Or with custom config
slm serve --config my-model.yaml --port 8000

API Usage

# Health check
curl http://localhost:8000/health

# Load model
curl -X POST http://localhost:8000/load \
  -H "Content-Type: application/json" \
  -d '{"config_path": "gpt2.yaml"}'

# Generate text
curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "The future of AI is",
    "max_tokens": 100,
    "temperature": 0.8
  }'

# Streaming
curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -H "Accept: text/event-stream" \
  -d '{"prompt": "Hello", "stream": true}'

---

📖 Documentation

Comprehensive guides for each component:

• Quick Start Guide - Complete walkthrough
• Model Formats Guide - GGUF vs PyTorch vs ONNX
• GGUF Setup Guide - Using llama.cpp with Metal/CUDA
• ONNX Guide - Export, run, and optimize ONNX models
• Init Guide - Creating configs manually
• Contributing Guide - Development setup and guidelines

---

🧪 Testing & Development

# Install dev dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Run with coverage
pytest --cov=slm_packager --cov-report=html

# Code quality
black slm_packager tests
isort slm_packager tests
mypy slm_packager

# Pre-commit hooks
pre-commit install
pre-commit run --all-files

Test Results:

• ✅ 73 tests passing
• Coverage: 52% overall
  • API: 82% ⭐
  • Core runtime: 60%
  • CLI: 47%

See CONTRIBUTING.md for detailed development guidelines.

---

🗺️ Roadmap

v0.2 (Current - December 2025)

• Automated test suite (✅ Complete: 73 tests, 52% coverage)
• MPS GPU support for Apple Silicon (✅ Complete: 2.14x speedup)
• ONNX runtime with KV-cache (✅ Complete: 13.8 tok/s)
• API server improvements (✅ Complete: 82% coverage)
• CUDA GPU acceleration testing
• Comprehensive benchmark suite
• Expand model registry

v1.0 (Future)

• vLLM integration for high-performance GPU serving
• ROCm support (AMD GPUs)
• Model conversion utilities
• Web UI for model management
• Advanced quantization options
• Multi-GPU support

---

🏗️ Architecture

┌─────────────────────────────────┐
│   CLI / API Server              │
├─────────────────────────────────┤
│   Model Registry & Downloader   │
├─────────────────────────────────┤
│   Runtime Abstraction Layer     │
│   ├─ llama.cpp (GGUF)           │
│   │   └─ Metal/CUDA support     │
│   ├─ Transformers (PyTorch)     │
│   │   └─ MPS/CUDA support       │
│   └─ ONNX Runtime               │
│       └─ Manual KV-cache        │
├─────────────────────────────────┤
│   Quantization & Benchmarking   │
└─────────────────────────────────┘

---

💡 Why SLM Packager?

Problem: Running small language models involves juggling different formats (GGUF, PyTorch, ONNX), runtimes (llama.cpp, transformers, onnxruntime), and configuration options.

Solution: SLM Packager provides:

• Unified interface - One CLI/API for all runtimes
• Auto-configuration - Models work out-of-the-box
• GPU acceleration - Automatic MPS on Mac, easy CUDA setup
• Reproducibility - YAML configs for deployment
• Developer-friendly - Python API, FastAPI server, streaming support

---

🤝 Contributing

We welcome contributions! See CONTRIBUTING.md for:

• Development setup
• Testing guidelines
• Code style requirements
• Pull request process

---

📝 License

MIT License - see LICENSE for details

---

🙏 Acknowledgments

• llama.cpp - Efficient GGUF runtime
• HuggingFace - Transformers and model hub
• ONNX Runtime - Optimized inference
• FastAPI - Modern API framework

---

📞 Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions
• GitHub: @Ayo-Cyber

---

Built with ❤️ for the AI community

Making small language models accessible, fast, and easy to deploy.