omni-local-llm 0.1.0

A unified Python interface and API server for running local Large Language Models.

One interface. Every local LLM.

A unified Python SDK and OpenAI-compatible API server for local large language models.

Quick Start • Features • Architecture • Installation • Usage • API Server • Contributing

---

Features

Feature	Description
Multi-Backend	Swap between Ollama and llama.cpp with a single parameter change
Chat Sessions	Built-in conversation memory and history management
Streaming	Real-time token-by-token streaming (sync & async)
Tool Calling	Native function/tool calling support across backends
JSON Mode	Enforce structured JSON output from any supported model
OpenAI-Compatible API	Drop-in FastAPI server compatible with the OpenAI SDK
Async-First	Full async/await support for high-concurrency workloads
Auto Model Pull	Automatically downloads models from Ollama or Hugging Face Hub

---

Quick Start

# Install
pip install omni-local-llm

# Or with uv (recommended)
uv pip install omni-local-llm

from omnillm import LocalLLMManager

manager = LocalLLMManager()
response = manager.chat(
    backend="ollama",
    model="llama3",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response)

That's it. Switch to llama.cpp by changing one word:

response = manager.chat(
    backend="llama.cpp",
    model="unsloth/llama-3-8b-Instruct-GGUF",
    messages=[{"role": "user", "content": "Hello!"}],
    filename="llama-3-8b-Instruct-Q4_K_M.gguf"
)

---

Architecture

┌─────────────────────────────────────────────────────────┐
│                      Your Application                   │
├─────────────┬─────────────────────┬─────────────────────┤
│   Python    │   Interactive CLI   │  OpenAI-Compatible  │
│     SDK     │  (python -m omnillm)│   FastAPI Server    │
├─────────────┴─────────────────────┴─────────────────────┤
│                    ChatSession                          │
│              (History · Streaming · Tools)              │
├─────────────────────────────────────────────────────────┤
│                  LocalLLMManager                        │
│              (Backend Registry & Router)                │
├────────────────────────┬────────────────────────────────┤
│    OllamaAdapter       │       LlamaCPPAdapter          │
│  (ollama Python SDK)   │  (llama-cpp-python + HF Hub)  │
└────────────────────────┴────────────────────────────────┘

Module Map

Module	Purpose
omnillm/core/base.py	LLMBackend — abstract base class for all adapters
omnillm/core/manager.py	LocalLLMManager — backend registry & request router
omnillm/core/session.py	ChatSession — conversation state, streaming, tool calls
omnillm/adapters/ollama_adapter.py	Ollama integration via official Python client
omnillm/adapters/llamacpp_adapter.py	llama.cpp integration + automatic GGUF model caching
omnillm/server.py	OpenAI-compatible FastAPI HTTP server
omnillm/__main__.py	Interactive CLI chat interface

---

Installation

Prerequisites

• Python 3.12+
• uv (recommended) or pip
• At least one backend:
  • Ollama installed and running, or
  • A GGUF model on Hugging Face Hub (auto-downloaded)

Install from source

git clone https://github.com/matteoespo/omni-local-llm.git
cd omni-local-llm
uv pip install -e .

Install from PyPI

pip install omni-local-llm
# or
uv pip install omni-local-llm

---

Usage

Python SDK

Basic Chat

from omnillm import LocalLLMManager

manager = LocalLLMManager()

response = manager.chat(
    backend="ollama",
    model="llama3",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response)

Chat Session (with Memory)

session = manager.create_session(
    backend="ollama",
    model="llama3",
    system_prompt="You are a helpful assistant."
)

response1 = session.send("Hello, I am Bob.")
response2 = session.send("What is my name?")  # Remembers "Bob"

Async & Streaming

import asyncio
from omnillm import LocalLLMManager

async def main():
    manager = LocalLLMManager()
    session = manager.create_session(backend="ollama", model="llama3")

    stream = await session.asend("Tell me a short story.", stream=True)
    async for chunk in stream:
        print(chunk, end="", flush=True)

asyncio.run(main())

Tool Calling

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {
                "city": {"type": "string", "description": "City name"}
            },
            "required": ["city"]
        }
    }
}]

response = manager.chat(
    backend="ollama",
    model="llama3",
    messages=[{"role": "user", "content": "What's the weather in Tokyo?"}],
    tools=tools
)
# response["tool_calls"] contains the function call

JSON Mode

response = manager.chat(
    backend="ollama",
    model="llama3",
    messages=[{"role": "user", "content": "List 3 colors as JSON"}],
    json_mode=True
)
# Returns valid JSON string

---

Interactive CLI

Start an interactive chat session directly in your terminal:

# Using Ollama
python -m omnillm --backend ollama --model llama3

# Using llama.cpp (auto-downloads from Hugging Face)
python -m omnillm --backend llama.cpp \
    --model unsloth/llama-3-8b-Instruct-GGUF \
    --filename llama-3-8b-Instruct-Q4_K_M.gguf

# With a system prompt
python -m omnillm --backend ollama --model llama3 \
    --system "You are a pirate. Respond only in pirate speak."

---

API Server

Launch an OpenAI-compatible local API server:

python -m omnillm.server
# Server runs on http://localhost:8000

Use with the OpenAI SDK

import openai

client = openai.OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed"
)

response = client.chat.completions.create(
    model="ollama/llama3",          # prefix with backend name
    messages=[{"role": "user", "content": "Explain relativity in one sentence."}]
)
print(response.choices[0].message.content)

Use with curl

curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ollama/llama3",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

---

Running Tests

# With uv (recommended)
uv run pytest

# Or activate the venv first
source .venv/bin/activate
pytest

# With coverage
uv run pytest --cov=omnillm --cov-report=term-missing

---

Contributing

Contributions are welcome! Please see the Contributing Guide for details on:

• Setting up your development environment
• Adding new LLM backend adapters
• Submitting pull requests

---

License

This project is licensed under the MIT License — see the LICENSE file for details.

---

Roadmap

• vLLM backend adapter
• KoboldCpp backend adapter
• Token usage tracking and reporting
• Vision/multimodal support
• Model hot-swapping in sessions
• Prompt template management
• Docker image for the API server

---

Made with ❤️ by matteoespo