ai-launcher-cli 0.1.0

Simple CLI to chat with GGUF models locally (no Ollama/LM Studio required)

GGUF Chat CLI

Simple Python CLI to chat with .gguf models locally using ctransformers. No Ollama, llama.cpp, LM Studio, or online providers required.

Install

pip install -r requirements.txt

Or install the package directly:

pip install -e .

Then use the ailaunch command:

ailaunch path/to/model.gguf

Usage

# Basic usage
python chat.py path/to/model.gguf

# Or installed as CLI
ailaunch path/to/model.gguf

# With custom settings
ailaunch model.gguf -c 8192 -t 0.8 --max-tokens 1024

# Disable streaming (wait for full response)
ailaunch model.gguf --no-stream

# Custom system prompt
ailaunch model.gguf --system "You are a coding assistant."

# Options:
#   -c, --ctx-size      Context window size (default: 4096)
#   -g, --gpu-layers    GPU layers to offload (-1 = all, default: -1)
#   -t, --threads       CPU threads (0 = auto, default: 0)
#   --temperature       Sampling temperature (default: 0.7)
#   --max-tokens        Max tokens to generate (default: 512)
#   --no-stream         Disable streaming output
#   --system            Custom system prompt

GPU Acceleration

Install with GPU extras for acceleration:

# NVIDIA CUDA
pip install ctransformers[cuda]

# Apple Metal (M1/M2/M3)
pip install ctransformers[metal]

Then use -g -1 to offload all layers to GPU.

Requirements

• Python 3.8+
• ctransformers (installs automatically via pip)

Exit

Type exit, quit, q, or press Ctrl+C to exit.