pyllama-server

Python wrapper for llama.cpp server - OpenAI API compatible backend with automatic device detection and model downloading.

Features

• Auto Device Detection: Automatically detects and configures the best GPU backend (Vulkan, CUDA, ROCm, Metal)
• Model Downloading: Download GGUF models from HuggingFace and ModelScope
• OpenAI API Compatible: Drop-in replacement for OpenAI API
• Function Calling: Support for tools and function calling
• Pre-built Binaries: Automatically downloads pre-built llama.cpp binaries

Installation

pip install pyllama-server

Quick Start

Command Line

# List available GPUs
pyllama devices

# Run inference with a model
pyllama run ./model.gguf -p "Hello, world!"

# Download a model
pyllama download llama-3.2-3b -q Q4_K_M

# Start an OpenAI-compatible server
pyllama serve llama-3.2-3b -p 8080

Python API

from pyllama import quick_run, quick_server, Client

# Quick inference
result = quick_run("llama-3.2-3b", "Write a haiku about coding")
print(result)

# Start server with auto-configuration
with quick_server("llama-3.2-3b") as server:
    client = Client(server.base_url)
    
    response = client.chat.completions.create(
        model="llama",
        messages=[{"role": "user", "content": "Hello!"}]
    )
    print(response.choices[0]["message"]["content"])

Auto Device Detection

from pyllama import DeviceDetector, AutoRunner

# Detect available GPUs
detector = DeviceDetector()
devices = detector.detect()
for device in devices:
    print(f"{device.name} ({device.backend.value}, {device.memory_gb:.1f}GB)")

# Get optimal configuration
config = detector.get_best_device(model_size_gb=5.0)
print(f"Best device: {config.device.name}")
print(f"Recommended GPU layers: {config.n_gpu_layers}")

Model Download

from pyllama import ModelDownloader

downloader = ModelDownloader()

# Download from HuggingFace
path = downloader.download("Qwen/Qwen2.5-7B-Instruct-GGUF", "Q4_K_M.gguf")

# Download from ModelScope
path = downloader.download(
    "LLM-Research/Meta-Llama-3-8B-Instruct-GGUF",
    "Meta-Llama-3-8B-Instruct-Q4_K_M.gguf",
    source="modelscope"
)

CLI Commands

Popular Models

GPU Backends

Function Calling

from pyllama import Client, LlamaServer

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get weather for a location",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {"type": "string"}
            }
        }
    }
}]

with LlamaServer("model.gguf") as server:
    client = Client(server.base_url)
    
    response = client.chat.completions.create(
        model="llama",
        messages=[{"role": "user", "content": "What's the weather in Tokyo?"}],
        tools=tools
    )
    
    if response.choices[0]["message"].get("tool_calls"):
        print("Model wants to call:", response.choices[0]["message"]["tool_calls"])

Requirements

• Python 3.8+
• Vulkan SDK (for Vulkan backend)
• CUDA Toolkit (for CUDA backend)
• ROCm (for ROCm backend)

License

MIT License - same as llama.cpp

Links

• llama.cpp
• Documentation
• Issues