aurestral 1.0.0

Local GGUF AI inference library built on llama-cpp-python with hardware auto-tuning

Aurestral

Local GGUF inference for Python, powered by llama-cpp-python. Aurestral discovers models in your project’s models/ folder, auto-tunes thread counts, context size, and GPU offload for your hardware, and ships with an interactive chatbot CLI.

Requirements

• Python 3.9+
• A GGUF model file (e.g. from Hugging Face)

Installation

pip install aurestral

For NVIDIA GPU acceleration, install llama-cpp-python with CUDA support first, then Aurestral:

# Windows / Linux (CUDA)
set CMAKE_ARGS=-DGGML_CUDA=on
set FORCE_CMAKE=1
pip install llama-cpp-python --force-reinstall --no-cache-dir

pip install aurestral

On macOS, the default llama-cpp-python wheel typically includes Metal acceleration.

Project layout

Place GGUF files in a models/ directory at your project root (or set AURESTRAL_MODELS_DIR):

my-project/
├── models/
│   └── llama-3.2-3b-instruct.Q4_K_M.gguf
└── main.py

Quick start

Interactive chatbot

cd my-project
aurestral
# or explicitly:
aurestral chat -m llama-3.2-3b-instruct.Q4_K_M.gguf

Chat commands: /help, /clear, /exit

Python API

from aurestral import load_model, ChatSession, generate

# One-shot completion
text = generate("Explain quantum entanglement in one sentence.")
print(text)

# Reusable model handle
model = load_model()  # auto-picks sole GGUF, or pass name="my-model"
reply = model.chat([
    {"role": "user", "content": "Hello!"},
])
print(reply)

# Multi-turn session with streaming
session = ChatSession.create(system_prompt="You are a concise coding assistant.")
session.send("Write a Python hello world.", stream=True)

List models and hardware info

aurestral list
aurestral info
aurestral run "The capital of France is" --stream

Hardware auto-tuning

On load, Aurestral inspects CPU cores, RAM, and whether llama-cpp-python was built with GPU offload support. It sets:

Setting	Behavior
n_threads	Physical cores minus one
n_ctx	1k–8k based on available RAM
n_gpu_layers	-1 (all layers) when GPU offload is available
use_mlock	Enabled on high-RAM CPU-only setups
flash_attn	Enabled when GPU offload is available

Override defaults with InferenceConfig or auto_tune=False:

from aurestral import InferenceConfig, load_model

cfg = InferenceConfig(n_ctx=8192, n_gpu_layers=35)
model = load_model("my-model.gguf", config=cfg, auto_tune=False)

Configuration reference

Environment

• AURESTRAL_MODELS_DIR — path to models folder (instead of ./models)

InferenceConfig — load-time: n_ctx, n_batch, n_threads, n_gpu_layers, use_mmap, use_mlock, flash_attn

GenerateConfig — generation-time: max_tokens, temperature, top_p, top_k, repeat_penalty, stop, stream

Publishing to PyPI

pip install build twine
python -m build
twine upload dist/*

License

MIT License — see LICENSE.