uc-llm-provider

Generic LLM provider abstraction for Python.
Anthropic, OpenAI, Google — and any OpenAI-compatible local model (Ollama, LM Studio, vLLM, llama.cpp).

Three usage modes: Library · Server · CLI

Install

pip install uc-llm-provider                  # core only
pip install "uc-llm-provider[anthropic]"     # + Anthropic SDK
pip install "uc-llm-provider[openai]"        # + OpenAI SDK
pip install "uc-llm-provider[server,cli]"    # + Server + CLI
pip install "uc-llm-provider[all]"           # everything

Library

from uc_llm_provider import get_provider, ChatRequest, ChatMessage

# Cloud
provider = get_provider({
    "name":          "anthropic",
    "provider_type": "anthropic",
    "api_key":       "sk-ant-...",
    "default_model": "claude-sonnet-4-20250514",
})

# Local (Ollama)
provider = get_provider({
    "name":          "ollama",
    "provider_type": "ollama",
    "api_base":      "http://localhost:11434",
    "default_model": "llama3.2",
})

# Any OpenAI-compatible endpoint
provider = get_provider({
    "name":          "lm-studio",
    "provider_type": "openai_compatible",
    "api_base":      "http://localhost:1234/v1",
    "default_model": "local-model",
})

# Chat
response = await provider.chat(ChatRequest(
    messages=[ChatMessage(role="user", content="Hello")],
    max_tokens=256,
))
print(response.content)

Streaming

async for chunk in provider.chat_stream(request):
    if chunk.type == "content_delta":
        print(chunk.content, end="", flush=True)

Custom provider

from uc_llm_provider import register_provider
from uc_llm_provider.providers.template import TemplateProvider

class MyProvider(TemplateProvider):
    def _get_endpoint(self, path=""):
        return "https://api.myprovider.com/v1/chat"

register_provider("myprovider", MyProvider)

Server

uc-llm-server --port 12290 --provider anthropic --model claude-sonnet-4-20250514
uc-llm-server --port 12290 --provider ollama --api-base http://localhost:11434 --model llama3.2
uc-llm-server --port 12290 --config server.yaml

Endpoints:

Config YAML:

provider: anthropic
model: claude-sonnet-4-20250514
api_key: ""          # or set UC_LLM_API_KEY
log_mode: jsonl      # jsonl | none
log_dir: ./logs
host: 0.0.0.0
cors_origins: ["*"]

CLI

uc-llm chat "Explain Playwright in one sentence"
uc-llm chat --interactive
uc-llm stream "Explain Playwright"
uc-llm models --provider ollama
uc-llm tokens "How many tokens is this?"
uc-llm health --provider anthropic
uc-llm config

Global options:

Provider types

Ollama extras

from uc_llm_provider import get_provider

ollama = get_provider({"provider_type": "ollama", "api_base": "http://localhost:11434"})
models = await ollama.list_local_models()
async for status in ollama.pull_model("mistral"):
    print(status)
running = await ollama.is_running()

License

MIT · cuber IT service