Pocket-O-Llama

Your local, high-precision GGUF model server.
Designed to run smoothly on low-spec hardware without throttling your system.
Empowering developers to run AI locally, efficiently, and privately.

Why Pocket-O-Llama?

Running local LLMs shouldn't require a supercomputer or a massive, bloated installation. Pocket-O-Llama is a lightweight, hardware-agnostic alternative to heavier engines. It is designed from the ground up to maximize the performance of highly quantized models (like those from Unsloth AI) on everyday laptops and desktops.

• Native Hugging Face Hub Integration: Pass any GGUF repository ID (e.g., unsloth/Llama-3.2-1B-Instruct-GGUF), and the server will auto-resolve, download, and cache the optimal quantization for you.
• True Zero-Copy Streaming: Real-time token streaming using llama_cpp for instant Time-To-First-Token.
• Interactive Web Dashboard: Comes with a beautiful, built-in dark-mode UI with Markdown rendering, model switching, and real-time conversation history.
• OpenAI Drop-In Replacement: Fully schema-compatible with /v1/chat/completions, meaning it instantly works with your existing LangChain, LlamaIndex, or AutoGen scripts.
• Zero Port Conflicts: Runs independently on custom ports (default 11435) so it never clashes with other local services.

Installation

Install Pocket-O-Llama directly via pip. It is recommended to use a virtual environment.

pip install pocket-o-llama

or

Clone the repository and install it locally via pip:

cd Pocket-O-Llama
python -m pip install .

Usage

Launch your local GGUF model server directly from the command line by passing the path to your model file:

# Example 1: Auto-download and run an Unsloth model from Hugging Face
pocket-chat --model unsloth/Llama-3.2-1B-Instruct-GGUF

# Example 2: Run a local .gguf file with custom thread allocation
pocket-chat --model "/path/to/your/model.gguf" --port 11435 --threads 4

Once running, open your browser and navigate to http://localhost:11435 to access the interactive chat dashboard!

Command-Line Arguments

API Usage/Testing

Pocket-O-Llama acts as a drop-in replacement for OpenAI/Gemini/Anthropic API. You can hit the /v1/chat/completions endpoint exactly as you normally would.

Using Python requests:

import requests

response = requests.post(
    "http://localhost:11435/v1/chat/completions",
    json={
        "messages": [
            {"role": "system", "content": "You are a helpful, brilliant coding assistant."},
            {"role": "user", "content": "Write a Python function to calculate the Fibonacci sequence."}
        ]
    }
)

print(response.json()["choices"][0]["message"]["content"])

Using cURL (Streaming):

curl -X POST http://localhost:11435/api/chat \
-H "Content-Type: application/json" \
-d '{
  "model": "unsloth/Llama-3.2-1B-Instruct-GGUF",
  "messages": [{"role": "user", "content": "Explain quantum computing in one sentence."}],
  "stream": true
}'

Roadmap (v0.2.0 & Beyond)

We are actively developing features to push the boundaries of lightweight local AI:

1. The "OS-Sandbox" HITL Agent: Zero-config native tool calling with interactive macOS-style GUI permission prompts (Human-in-the-Loop) before executing code or reading local directories.

2. Dynamic LoRA Swapping: Instant, memory-efficient "Skill Cartridge" swapping for base models.

3. LAN Compute Swarming: Peer discovery to split GGUF inference layers across multiple low-spec devices on the same Wi-Fi network.

License & Author

Created by Hrutu Surve

This project is open-source and fiercely protected against unauthorized closed-source corporate usage under the GNU AGPLv3 License. See the LICENSE file for full details.