omni-local-llm 0.2.0

One interface. Every local LLM.

A unified Python SDK and OpenAI-compatible API server for local large language models.

Quick Start • Features • Architecture • Installation • Usage • API Server • Contributing

Features

Quick Start

# Install
pip install "omni-local-llm[ollama,server]"

# Or with uv (recommended)
uv pip install "omni-local-llm[ollama,server]"

from omnillm import LocalLLMManager

manager = LocalLLMManager()
response = manager.chat(
    backend="ollama",
    model="llama3",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.content)

That's it. Switch to llama.cpp by changing one word:

response = manager.chat(
    backend="llama.cpp",
    model="unsloth/llama-3-8b-Instruct-GGUF",
    messages=[{"role": "user", "content": "Hello!"}],
    filename="llama-3-8b-Instruct-Q4_K_M.gguf"
)

Architecture

┌─────────────────────────────────────────────────────────┐
│                      Your Application                   │
├─────────────┬─────────────────────┬─────────────────────┤
│   Python    │   Interactive CLI   │  OpenAI-Compatible  │
│     SDK     │  (python -m omnillm)│   FastAPI Server    │
├─────────────┴─────────────────────┴─────────────────────┤
│                    ChatSession                          │
│              (History · Streaming · Tools)              │
├─────────────────────────────────────────────────────────┤
│                  LocalLLMManager                        │
│              (Backend Registry & Router)                │
├────────────────────────┬────────────────────────────────┤
│    OllamaAdapter       │       LlamaCPPAdapter          │
│  (ollama Python SDK)   │  (llama-cpp-python + HF Hub)  │
└────────────────────────┴────────────────────────────────┘

Module Map

Installation

Prerequisites

• Python 3.12+
• uv (recommended) or pip
• At least one backend:
  • Ollama installed and running, or
  • A GGUF model on Hugging Face Hub (auto-downloaded)

Install from source

git clone https://github.com/matteoespo/omni-local-llm.git
cd omni-local-llm
uv pip install -e ".[ollama,server]"

Install from PyPI

pip install "omni-local-llm[ollama,server]"
# or
uv pip install "omni-local-llm[ollama,server]"

Usage

Python SDK

Basic Chat

from omnillm import LocalLLMManager

manager = LocalLLMManager()

response = manager.chat(
    backend="ollama",
    model="llama3",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.content)

Chat Session (with Memory & Context Window Pruning)

ChatSession maintains conversation history, commits turns only on success, and supports proactive sliding-window context pruning to keep local models within their context limits:

# Create a session with turn and token budget constraints
session = manager.create_session(
    backend="ollama",
    model="llama3",
    system_prompt="You are a helpful coding assistant.",
    max_turns=10,             # Keep the most recent 10 conversational turns
    max_tokens_budget=4096,   # Automatically evict oldest turns before exceeding context budget
    strategy="sliding_window",
)

# System prompt is ALWAYS preserved during pruning
response1 = session.send("Hello, I am Bob.")
response2 = session.send("What is my name?")  # Remembers "Bob"

# Inspection properties
print(f"Stored turns: {session.turn_count}")
print(f"Estimated token footprint: {session.estimated_tokens}")

# Reset / clear controls
session.clear()                # Clears history, keeping original system_prompt
session.reset("New prompt")    # Clears history and updates system prompt

Async & Streaming

import asyncio
from omnillm import LocalLLMManager

async def main():
    manager = LocalLLMManager()
    session = manager.create_session(backend="ollama", model="llama3")

    stream = await session.asend("Tell me a short story.", stream=True)
    async for chunk in stream:
        print(chunk, end="", flush=True)

asyncio.run(main())

Tool Calling

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {
                "city": {"type": "string", "description": "City name"}
            },
            "required": ["city"]
        }
    }
}]

response = manager.chat(
    backend="ollama",
    model="llama3",
    messages=[{"role": "user", "content": "What's the weather in Tokyo?"}],
    tools=tools
)
# response.tool_calls contains the function call

Structured Outputs & Pydantic Validation

Guarantee JSON schema adherence at the sampler level (using Ollama schemas or llama.cpp GBNF grammars) and receive typed Pydantic models automatically:

from pydantic import BaseModel

class UserProfile(BaseModel):
    name: str
    age: int
    skills: list[str]

response = manager.chat(
    backend="ollama",
    model="llama3",
    messages=[{"role": "user", "content": "Extract: Alice is a 28-year-old engineer skilled in Python and Rust."}],
    response_model=UserProfile,
)

# Access validated Pydantic object directly
user: UserProfile = response.parsed
print(user.name)    # Alice
print(user.skills)  # ['Python', 'Rust']

You can also pass raw JSON schemas or request basic JSON mode:

# Raw JSON Schema
response = manager.chat(
    backend="ollama",
    model="llama3",
    messages=[{"role": "user", "content": "Generate a city record"}],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "City",
            "schema": {
                "type": "object",
                "properties": {"city": {"type": "string"}, "population": {"type": "integer"}},
                "required": ["city", "population"],
            },
        },
    },
)

# Basic JSON Mode
response = manager.chat(
    backend="ollama",
    model="llama3",
    messages=[{"role": "user", "content": "List 3 colors as JSON"}],
    json_mode=True,
)

Embeddings (RAG & Semantic Search)

Generate dense vector embeddings for documents or search queries using either backend:

# Synchronous embeddings with Ollama
response = manager.embed(
    backend="ollama",
    model="nomic-embed-text",
    input=["Retrieval-augmented generation with local LLMs", "Semantic similarity search"]
)
print(response.embeddings)  # [[0.021, -0.043, ...], [0.112, 0.009, ...]]
print(response.usage.prompt_tokens)

# Embeddings with llama.cpp (auto-downloads GGUF from Hugging Face Hub)
response = manager.embed(
    backend="llama.cpp",
    model="nomic-ai/nomic-embed-text-v1.5-GGUF",
    filename="nomic-embed-text-v1.5.Q4_K_M.gguf",
    input="Single string input is also supported"
)

# Asynchronous embeddings
response = await manager.aembed(
    backend="ollama",
    model="nomic-embed-text",
    input=["Async batch embedding"]
)

Vision & Multimodal (Image Understanding)

Run local multimodal models (such as llava, llama3.2-vision, or qwen2-vl) using local file paths, raw bytes, base64 data URIs, or OpenAI-standard image parts:

# Direct image input via file path or bytes (Ollama)
response = manager.chat(
    backend="ollama",
    model="llava",
    messages=[{
        "role": "user",
        "content": "What is depicted in this photo?",
        "images": ["photo.jpg"],  # accepts file paths, Path objects, base64, or bytes
    }]
)
print(response.content)

# Using llama.cpp with Hugging Face GGUF + multimodal projector (mmproj)
response = manager.chat(
    backend="llama.cpp",
    model="myself/llava-1.5-7b-GGUF",
    filename="llava-1.5-7b-Q4_K.gguf",
    mmproj_filename="mmproj-model-f16.gguf",  # automatically downloaded & loaded
    messages=[{
        "role": "user",
        "content": "Analyze this chart.",
        "images": ["chart.png"],
    }]
)

# In ChatSession with conversational memory
session = manager.create_session(backend="ollama", model="llava")
response = session.send("What is in this image?", images=["invoice.png"])
response2 = session.send("Extract the total amount.")  # Remembers the image context!

# OpenAI-compatible multi-part format (also supported via FastAPI server)
response = manager.chat(
    backend="ollama",
    model="llava",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this image in detail:"},
            {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
        ]
    }]
)

Interactive CLI

Start an interactive chat session directly in your terminal:

# Using Ollama
python -m omnillm --backend ollama --model llama3

# Using llama.cpp (auto-downloads from Hugging Face)
python -m omnillm --backend llama.cpp \
    --model unsloth/llama-3-8b-Instruct-GGUF \
    --filename llama-3-8b-Instruct-Q4_K_M.gguf

# With a system prompt
python -m omnillm --backend ollama --model llama3 \
    --system "You are a pirate. Respond only in pirate speak."

API Server

Launch an OpenAI-compatible local API server:

python -m omnillm.server
# Server runs on http://localhost:8000

Use with the OpenAI SDK

import openai

client = openai.OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed"
)

# Chat completions
response = client.chat.completions.create(
    model="ollama/llama3",          # prefix with backend name
    messages=[{"role": "user", "content": "Explain relativity in one sentence."}]
)
print(response.choices[0].message.content)

# Structured Outputs (JSON Schema)
structured_res = client.chat.completions.create(
    model="ollama/llama3",
    messages=[{"role": "user", "content": "Extract: Alice, age 29"}],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "User",
            "schema": {
                "type": "object",
                "properties": {"name": {"type": "string"}, "age": {"type": "integer"}},
                "required": ["name", "age"],
            },
        },
    },
)
print(structured_res.choices[0].message.content)

# Embeddings
embedding_res = client.embeddings.create(
    model="ollama/nomic-embed-text",
    input=["Embed this chunk for vector search", "Another document"]
)
print(embedding_res.data[0].embedding)

Use with curl

# Chat completions
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ollama/llama3",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

# Structured Outputs with JSON Schema
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ollama/llama3",
    "messages": [{"role": "user", "content": "Alice is 29 years old."}],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "User",
        "schema": {
          "type": "object",
          "properties": {"name": {"type": "string"}, "age": {"type": "integer"}},
          "required": ["name", "age"]
        }
      }
    }
  }'

# Embeddings
curl -X POST http://localhost:8000/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ollama/nomic-embed-text",
    "input": ["Local vector embeddings with Omni-Local-LLM"]
  }'

Running Tests

# With uv (recommended)
uv run pytest

# Or activate the venv first
source .venv/bin/activate
pytest

# With coverage
uv run pytest --cov=omnillm --cov-report=term-missing

# Type checking
uv run mypy omnillm

Contributing

Contributions are welcome! Please see the Contributing Guide for details on:

• Setting up your development environment
• Adding new LLM backend adapters
• Submitting pull requests

License

This project is licensed under the MIT License — see the LICENSE file for details.

Roadmap

• vLLM backend adapter
• KoboldCpp backend adapter
• Token usage tracking and reporting
• Vision/multimodal support
• Model hot-swapping in sessions
• Prompt template management
• Docker image for the API server

Made with ❤️ by matteoespo