liter-llm 1.4.0rc2

Universal LLM API client for Python. Unified interface for streaming, tool calling, and provider routing across 142+ LLM providers. Rust-powered.

Python

Universal LLM API client for Python. Access 143+ LLM providers through a single unified interface. Native async/await support, streaming responses, tool calling, and type-safe API.

Installation

Package Installation

Install via pip:

pip install liter-llm

System Requirements

• Python 3.10+ required
• API keys via environment variables (e.g. OPENAI_API_KEY, ANTHROPIC_API_KEY)

Quick Start

Basic Chat

Send a message to any provider using the provider/model prefix:

import asyncio
import os
from liter_llm import LlmClient

async def main() -> None:
    client = LlmClient(api_key=os.environ["OPENAI_API_KEY"])
    response = await client.chat(
        model="openai/gpt-4o",
        messages=[{"role": "user", "content": "Hello!"}],
    )
    print(response.choices[0].message.content)

asyncio.run(main())

Common Use Cases

Streaming Responses

Stream tokens in real time:

import asyncio
import os
from liter_llm import LlmClient

async def main() -> None:
    client = LlmClient(api_key=os.environ["OPENAI_API_KEY"])
    async for chunk in await client.chat_stream(
        model="openai/gpt-4o",
        messages=[{"role": "user", "content": "Tell me a story"}],
    ):
        if chunk.choices and chunk.choices[0].delta.content:
            print(chunk.choices[0].delta.content, end="", flush=True)
    print()

asyncio.run(main())

Tool Calling

Define and invoke tools:

import asyncio
import os
from liter_llm import LlmClient

async def main() -> None:
    client = LlmClient(api_key=os.environ["OPENAI_API_KEY"])

    tools = [
        {
            "type": "function",
            "function": {
                "name": "get_weather",
                "description": "Get the current weather for a location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {"type": "string", "description": "City name"},
                    },
                    "required": ["location"],
                },
            },
        }
    ]

    response = await client.chat(
        model="openai/gpt-4o",
        messages=[{"role": "user", "content": "What is the weather in Berlin?"}],
        tools=tools,
    )

    choice = response.choices[0]
    if choice.message.tool_calls:
        for call in choice.message.tool_calls:
            print(f"Tool: {call.function.name}, Args: {call.function.arguments}")

asyncio.run(main())

Next Steps

• Provider Registry - Full list of supported providers
• GitHub Repository - Source, issues, and discussions

Features

Supported Providers (143+)

Route to any provider using the provider/model prefix convention:

Provider	Example Model
OpenAI	openai/gpt-4o, openai/gpt-4o-mini
Anthropic	anthropic/claude-3-5-sonnet-20241022
Groq	groq/llama-3.1-70b-versatile
Mistral	mistral/mistral-large-latest
Cohere	cohere/command-r-plus
Together AI	together/meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo
Fireworks	fireworks/accounts/fireworks/models/llama-v3p1-70b-instruct
Google Vertex	vertexai/gemini-1.5-pro
Amazon Bedrock	bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0

Complete Provider List

Key Capabilities

• Provider Routing -- Single client for 143+ LLM providers via provider/model prefix

• Local LLMs — Connect to locally-hosted models via Ollama, LM Studio, vLLM, llama.cpp, and other local inference servers

• Unified API -- Consistent chat, chat_stream, embeddings, list_models interface

• Streaming -- Real-time token streaming via chat_stream

• Tool Calling -- Function calling and tool use across all supporting providers

• Type Safe -- Schema-driven types compiled from JSON schemas

• Secure -- API keys never logged or serialized, managed via environment variables

• Observability -- Built-in OpenTelemetry with GenAI semantic conventions

• Error Handling -- Structured errors with provider context and retry hints

Performance

Built on a compiled Rust core for speed and safety:

• Provider resolution at client construction -- zero per-request overhead
• Configurable timeouts and connection pooling
• Zero-copy streaming with SSE and AWS EventStream support
• API keys wrapped in secure memory, zeroed on drop

Provider Routing

Route to 143+ providers using the provider/model prefix convention:

openai/gpt-4o
anthropic/claude-3-5-sonnet-20241022
groq/llama-3.1-70b-versatile
mistral/mistral-large-latest

See the provider registry for the full list.

Proxy Server

liter-llm also ships as an OpenAI-compatible proxy server with Docker support:

docker run -p 4000:4000 -e LITER_LLM_MASTER_KEY=sk-your-key ghcr.io/kreuzberg-dev/liter-llm

See the proxy server documentation for configuration, CLI usage, and MCP integration.

Documentation

• Documentation -- Full docs and API reference
• GitHub Repository -- Source, issues, and discussions
• Provider Registry -- 143 supported providers

Part of kreuzberg.dev.

Contributing

Contributions are welcome! See CONTRIBUTING.md for guidelines.

Join our Discord community for questions and discussion.

License

MIT -- see LICENSE for details.