llmai 0.3.5

Unified Python client for OpenAI, Azure OpenAI, Vertex AI, Anthropic, Gemini, DeepSeek, OpenRouter, Cerebras, Fireworks, Together AI, LM Studio, Bedrock, LiteLLM, and ChatGPT.

llmai

llmai is a Python library for working with OpenAI, Azure OpenAI, Vertex AI, Anthropic, Google Gemini, DeepSeek, OpenRouter, Cerebras, Fireworks, Together AI, LM Studio, Bedrock, LiteLLM, and ChatGPT through a shared set of message, tool, schema, and response primitives.

Today the repository includes adapters for:

• ChatGPT
• OpenAI
• Azure OpenAI
• Vertex AI
• DeepSeek
• OpenRouter
• Cerebras
• Fireworks
• Together AI
• LM Studio
• Anthropic
• Google Gemini
• Amazon Bedrock
• LiteLLM

Each synchronous provider client exposes the same core entrypoint:

• generate(..., stream=False)

Every provider also has a separate async client with:

• await agenerate(..., stream=False)
• async for ... in agenerate(..., stream=True)

Local Model Metadata

The complete models.dev/api.json response is bundled at llmai/data/models.json. It currently contains thousands of model records, including context limits, output limits, capabilities, modalities, pricing, and release metadata.

from llmai import (
    get_context_window,
    get_model_metadata,
    query_models,
    refresh_model_data,
)

# Provider-qualified references are unambiguous.
context = get_context_window("openai:gpt-4.1")
metadata = get_model_metadata("openai:gpt-4.1")
print(context)
print(metadata["max_output_tokens"], metadata["modalities"])

# Search names, IDs, references, families, and descriptions.
reasoning_models = query_models("reasoning", provider="anthropic")

# Re-download, validate, and atomically replace the bundled JSON.
refresh_model_data()

Each flattened record retains every upstream model field and includes provider_metadata for the provider-level API, documentation, environment, and package information. Bare model IDs and names are accepted when they identify exactly one record. When multiple providers expose the same ID, use provider:model-id, the provider= argument, or query_models() to inspect every match. Use load_model_data() when you need the original provider-keyed JSON structure, and list_models() for flattened records with context_window extracted. If an upstream record has no valid context limit, context_window and get_context_window() reliably fall back to 4000. An unknown or ambiguous model also returns this fallback from get_context_window(); pass default=... to customize it. get_model_metadata() remains strict and raises a lookup error when callers need to distinguish missing or ambiguous records.

Live Model Availability

The bundled metadata above describes known models; it does not prove that a specific API key, account, or self-hosted endpoint can access them. Use the live listing API for credential validation and model pickers:

import asyncio

from llmai import OpenAIClientConfig, alist_available_models


async def main():
    models = await alist_available_models(
        config=OpenAIClientConfig(
            api_key="<your-openai-api-key>",
        )
    )
    print(models)


asyncio.run(main())

Synchronous applications can call list_available_models(config=...). Configured provider clients also expose list_available_models(), and async clients expose await alist_available_models(). OpenAI, Anthropic, Google, DeepSeek, OpenRouter, Cerebras, Fireworks, Together AI, LM Studio, and LiteLLM use live provider APIs. An OpenAIClientConfig with a custom base_url also supports OpenAI-compatible services such as Ollama and vLLM.

Failures use the same normalized LLMAuthenticationError, LLMRateLimitError, LLMConnectionError, LLMConfigurationError, and LLMError classes as generation calls. Their status_code, message, and provider fields are suitable for API-layer translation. Providers without a live listing implementation raise LLMConfigurationError instead of silently falling back to the bundled metadata.

Why This Exists

Provider SDKs differ in how they represent messages, tool calls, structured output, and streaming events. llmai smooths those differences out so application code can stay closer to one mental model.

Installation

Install the project locally with uv:

uv sync

Or install it in editable mode with pip:

pip install -e .

Quick Start

from llmai import OpenAIClient, OpenAIClientConfig
from llmai.shared import UserMessage

client = OpenAIClient(
    config=OpenAIClientConfig(api_key="<your-openai-api-key>"),
)

result = client.generate(
    model="your-openai-model",
    messages=[
        UserMessage(content="Write a two-line poem about clean interfaces."),
    ],
)

print(result.content)
print(result.usage)
print(result.duration_seconds)

For text-only prompts, UserMessage(content="...") is the simplest form. SystemMessage(content="...") also takes a plain string. Use explicit content parts like TextContentPart only when you need mixed multimodal input or tighter control over user message structure.

If you want to swap providers, the overall call shape stays the same. In most cases you only need to change the client class, credentials, and model name.

Async Usage

Async clients are separate from their synchronous counterparts, so adding async support does not change existing applications:

import asyncio

from llmai import AsyncOpenAIClient, OpenAIClientConfig
from llmai.shared import UserMessage


async def main():
    async with AsyncOpenAIClient(
        config=OpenAIClientConfig(api_key="<your-openai-api-key>"),
    ) as client:
        result = await client.agenerate(
            model="your-openai-model",
            messages=[UserMessage(content="Explain async I/O briefly.")],
        )
        print(result.content)


asyncio.run(main())

Use get_async_client() when the provider is selected dynamically:

from llmai import OpenAIClientConfig, get_async_client

client = get_async_client(
    config=OpenAIClientConfig(api_key="<your-openai-api-key>"),
)

Independent calls can run concurrently:

first, second = await asyncio.gather(
    client.agenerate(model="your-model", messages=first_messages),
    client.agenerate(model="your-model", messages=second_messages),
)
await client.aclose()

OpenAI-compatible providers, Azure, Anthropic, Google, and Vertex use their provider SDK's native async transport. Bedrock uses asyncio.to_thread because boto3 does not provide an async client. This prevents Bedrock calls from blocking the event loop, although an already-running boto3 operation cannot be forcibly cancelled.

Azure OpenAI

from llmai import AzureOpenAIClient, AzureOpenAIClientConfig
from llmai.shared import UserMessage


client = AzureOpenAIClient(
    config=AzureOpenAIClientConfig(
        api_key="<your-azure-openai-api-key>",
        endpoint="https://your-resource.openai.azure.com",
        api_version="2024-10-21",
    ),
)

result = client.generate(
    model="your-azure-deployment",
    messages=[
        UserMessage(content="Write a two-line poem about clean interfaces."),
    ],
)

print(result.content)

AzureOpenAIClient uses the official OpenAI SDK's Azure client and requires an explicit AzureOpenAIClientConfig. The config supports API-key auth or Entra token auth and accepts endpoint or base_url, api_version, and optional deployment. Azure is always routed through chat completions.

Vertex AI

from llmai import VertexAIClient, VertexAIClientConfig
from llmai.shared import UserMessage


client = VertexAIClient(
    config=VertexAIClientConfig(
        project="your-gcp-project",
        location="us-central1",
    ),
)

result = client.generate(
    model="gemini-2.5-flash",
    messages=[
        UserMessage(content="Write a two-line poem about clean interfaces."),
    ],
)

print(result.content)

VertexAIClient uses the google-genai Vertex AI path internally and requires an explicit VertexAIClientConfig. Use either api_key for Vertex express mode or project/location/credentials for standard Vertex auth; do not combine them. base_url remains optional.

ChatGPT

from llmai import ChatGPTClient, ChatGPTClientConfig
from llmai.shared import UserMessage


client = ChatGPTClient(
    config=ChatGPTClientConfig(access_token="<your-chatgpt-access-token>"),
)

result = client.generate(
    model="chatgpt-4o-latest",
    messages=[
        UserMessage(content="Write a two-line poem about clean interfaces."),
    ],
)

print(result.content)

ChatGPTClient targets ChatGPT's Codex backend at https://chatgpt.com/backend-api/codex and always uses the Responses API internally. Credentials and optional overrides are passed through ChatGPTClientConfig, which uses access_token. When you include SystemMessage entries, ChatGPT collects them in order and sends them through the Responses API instructions field; otherwise it falls back to instructions="Follow the prompt". The ChatGPT backend requires stream=True, so generate(stream=False) streams internally and returns the aggregated final response. It also does not support Responses temperature or max_output_tokens, so temperature and max_tokens are ignored for this client.

DeepSeek

from llmai import DeepSeekClient, DeepSeekClientConfig
from llmai.shared import JSONSchemaResponse, UserMessage


client = DeepSeekClient(
    config=DeepSeekClientConfig(api_key="<your-deepseek-api-key>"),
)

result = client.generate(
    model="deepseek-chat",
    messages=[
        UserMessage(content="Return a JSON object with one field named answer."),
    ],
    response_format=JSONSchemaResponse(
        name="final_answer",
        json_schema={
            "type": "object",
            "properties": {
                "answer": {"type": "string"},
            },
            "required": ["answer"],
        },
    ),
)

print(result.content)

DeepSeekClient uses the OpenAI SDK against DeepSeek's OpenAI-compatible API and requires an explicit DeepSeekClientConfig. For structured output, it always uses an internal function-tool schema because DeepSeek does not support response_format={"type":"json_schema"}. During streaming, the internal response tool is surfaced as incremental JSON content chunks, and the stream still ends with parsed JSON on the final completion chunk's content. If you need DeepSeek's server-side strict tool enforcement, point base_url at https://api.deepseek.com/beta.

OpenRouter

from llmai import OpenRouterClient, OpenRouterClientConfig
from llmai.shared import UserMessage


client = OpenRouterClient(
    config=OpenRouterClientConfig(api_key="<your-openrouter-api-key>"),
)

result = client.generate(
    model="openai/gpt-5.4-mini",
    messages=[
        UserMessage(content="Write a two-line poem about clean interfaces."),
    ],
)

print(result.content)

OpenRouterClient uses the OpenAI SDK against OpenRouter's OpenAI-compatible chat-completions API. The default base URL is https://openrouter.ai/api/v1.

Fireworks

from llmai import FireworksClient, FireworksClientConfig
from llmai.shared import UserMessage


client = FireworksClient(
    config=FireworksClientConfig(api_key="<your-fireworks-api-key>"),
)

result = client.generate(
    model="accounts/fireworks/models/llama-v3p1-8b-instruct",
    messages=[
        UserMessage(content="Write a two-line poem about clean interfaces."),
    ],
)

print(result.content)

FireworksClient uses the OpenAI SDK against Fireworks' OpenAI-compatible chat-completions API. The default base URL is https://api.fireworks.ai/inference/v1.

Together AI

from llmai import TogetherAIClient, TogetherAIClientConfig
from llmai.shared import UserMessage


client = TogetherAIClient(
    config=TogetherAIClientConfig(api_key="<your-together-api-key>"),
)

result = client.generate(
    model="openai/gpt-oss-20b",
    messages=[
        UserMessage(content="Write a two-line poem about clean interfaces."),
    ],
)

print(result.content)

TogetherAIClient uses the OpenAI SDK against Together AI's OpenAI-compatible chat-completions API. The default base URL is https://api.together.ai/v1. Together AI does not support the OpenAI Responses API, so this adapter always uses chat completions.

LM Studio

from llmai import LMStudioClient, LMStudioClientConfig
from llmai.shared import UserMessage


client = LMStudioClient(
    config=LMStudioClientConfig(
        base_url="http://localhost:1234",
    ),
)

result = client.generate(
    model="openai/gpt-oss-20b",
    messages=[
        UserMessage(content="Write a two-line poem about clean interfaces."),
    ],
)

print(result.content)

LMStudioClient uses the OpenAI SDK against LM Studio's OpenAI-compatible chat-completions endpoint. The default base URL is http://localhost:1234/v1, custom base URLs have /v1 appended automatically when omitted, and api_key is optional. Schemas are reduced to LM Studio's grammar-friendly core fields, so unsupported regex pattern constraints are removed before sending.

Cerebras

from llmai import CerebrasClient, CerebrasClientConfig
from llmai.shared import UserMessage


client = CerebrasClient(
    config=CerebrasClientConfig(api_key="<your-cerebras-api-key>"),
)

result = client.generate(
    model="llama-4-scout-17b-16e-instruct",
    messages=[
        UserMessage(content="Write a two-line poem about clean interfaces."),
    ],
)

print(result.content)

CerebrasClient uses the OpenAI SDK against Cerebras' OpenAI-compatible API. The default base URL is https://api.cerebras.ai/v1.

Amazon Bedrock

from llmai import BedrockClient, BedrockClientConfig
from llmai.shared import UserMessage


client = BedrockClient(
    config=BedrockClientConfig(
        region="us-east-1",
        aws_access_key_id="<your-aws-access-key-id>",
        aws_secret_access_key="<your-aws-secret-access-key>",
    ),
)

# Or use Bedrock API-key auth:
# client = BedrockClient(
#     config=BedrockClientConfig(region="us-east-1", api_key="<your-bedrock-api-key>")
# )

result = client.generate(
    model="us.anthropic.claude-3-5-haiku-20241022-v1:0",
    messages=[
        UserMessage(content="Say hello."),
    ],
)

print(result.content)

LiteLLM

from llmai import LiteLLMClient, LiteLLMClientConfig
from llmai.shared import UserMessage


client = LiteLLMClient(
    config=LiteLLMClientConfig(
        api_key="litellm-proxy-key",
        base_url="https://litellm.example/v1",
    )
)

result = client.generate(
    model="gpt-5.4-mini",
    messages=[
        UserMessage(content="Write a two-line poem about clean interfaces."),
    ],
)

print(result.content)

LiteLLMClient uses the OpenAI Python client against an OpenAI-compatible LiteLLM proxy. Set LiteLLMClientConfig(api_type=OpenAIApiType.RESPONSES) to call the proxy's Responses API instead of chat completions. Pass the proxy key and URL through api_key and base_url; config-level extra_kwargs and per-call generate(..., extra_body={...}) are forwarded as request extra_body.

Structured Output

from pydantic import BaseModel

from llmai import GoogleClient, GoogleClientConfig
from llmai.shared import JSONSchemaResponse, UserMessage


class Summary(BaseModel):
    title: str
    bullets: list[str]


client = GoogleClient(config=GoogleClientConfig(api_key="<your-google-api-key>"))

result = client.generate(
    model="your-google-model",
    messages=[
        UserMessage(content="Summarize retrieval-augmented generation in simple terms."),
    ],
    response_format=JSONSchemaResponse(json_schema=Summary),
)

print(result.content)

Use JSONSchemaResponse, JSONObjectResponse, or TextResponse to request different response shapes.

Multimodal Content

from llmai import GoogleClient, GoogleClientConfig
from llmai.shared import ImageContentPart, TextContentPart, UserMessage


client = GoogleClient(config=GoogleClientConfig(api_key="<your-google-api-key>"))

result = client.generate(
    model="your-google-model",
    messages=[
        UserMessage(
            content=[
                TextContentPart(text="Describe this image."),
                ImageContentPart(url="https://example.com/cat.png"),
            ]
        ),
    ],
)

print(result.content)
print(result.thinking)

Use explicit content parts when you need multimodal inputs or want to mix text with images in one message. Normal completion content is surfaced as list[TextContentPart | ImageContentPart] when the provider returns message content, including text-only replies. Reasoning is exposed on ResponseContent.thinking as list[str] when the provider returns one or more thinking blocks, and the same value is also available on the final AssistantMessage.

Tool Calling

from pydantic import BaseModel

from llmai import OpenAIClient, OpenAIClientConfig
from llmai.shared import Tool, ToolResponseMessage, UserMessage


class WeatherArgs(BaseModel):
    city: str


weather_tool = Tool(
    name="get_weather",
    description="Look up the weather for a city.",
    schema=WeatherArgs,
)

client = OpenAIClient(config=OpenAIClientConfig(api_key="<your-openai-api-key>"))

first = client.generate(
    model="your-openai-model",
    messages=[
        UserMessage(content="What is the weather in Kathmandu?"),
    ],
    tools=[weather_tool],
    tool_choice={"tools": ["get_weather"]},
)

for tool_call in first.tool_calls:
    if tool_call.name != "get_weather":
        continue

    follow_up = client.generate(
        model="your-openai-model",
        messages=[
            *first.messages,
            ToolResponseMessage(
                id=tool_call.id,
                content=["It is sunny in Kathmandu."],
            ),
        ],
        tools=[weather_tool],
    )
    print(follow_up.content)

llmai returns tool calls in first.tool_calls and leaves execution to the caller.

Hosted Web Search

llmai also supports a provider-hosted web search tool that is not a function tool:

from llmai import OpenAIApiType, OpenAIClient, OpenAIClientConfig
from llmai.shared import UserMessage, WebSearchTool

client = OpenAIClient(
    config=OpenAIClientConfig(
        api_key="<your-openai-api-key>",
        api_type=OpenAIApiType.RESPONSES,
    )
)

result = client.generate(
    model="your-openai-model",
    messages=[
        UserMessage(content="What was a positive news story from today? Cite sources."),
    ],
    tools=[WebSearchTool()],
)

print(result.content)
print(result.thinking)

You can also target it explicitly in tool_choice:

tool_choice = {
    "mode": "required",
    "tools": ["web_search"],
}

Current llmai behavior:

• OpenAI Responses: attaches built-in web_search
• Azure OpenAI: follows the same OpenAI adapter surface; service support depends on your Azure API version and deployment
• Vertex AI: attaches google_search
• ChatGPT/Codex: attaches built-in web_search
• Anthropic: attaches Anthropic's hosted web-search tool
• Google Gemini: attaches google_search
• OpenAI Chat Completions: ignores hosted web_search
• DeepSeek: ignores hosted web_search
• Amazon Bedrock: ignores hosted web_search

web_search can be mixed with normal function tools in the same request.

If you use OpenAIClient with api_type=OpenAIApiType.RESPONSES, OpenAIClientConfig(provide_system_message_as_instructions=True) lifts all SystemMessage values into the top-level Responses API instructions field. The default is False, which keeps system messages inline in input.

Streaming

from llmai import AnthropicClient, AnthropicClientConfig
from llmai.shared import UserMessage

client = AnthropicClient(
    config=AnthropicClientConfig(api_key="<your-anthropic-api-key>"),
)

for chunk in client.generate(
    model="your-anthropic-model",
    messages=[
        UserMessage(content="Explain recursion in one paragraph."),
    ],
    stream=True,
):
    if chunk.type == "content":
        print(chunk.chunk, end="")
    elif chunk.type == "completion":
        print("\nDone:", chunk.usage)

generate(..., stream=True) yields marker chunks with type="event" and event="start" / event="end" around each content, thinking, and tool section. If a provider returns multiple reasoning blocks, each block gets its own thinking start/end pair. The final chunk has type="completion" and includes top-level content, thinking, usage, and accumulated messages.

Async clients expose the same chunks through direct async iteration:

from llmai import AsyncAnthropicClient, AnthropicClientConfig
from llmai.shared import UserMessage

async with AsyncAnthropicClient(
    config=AnthropicClientConfig(api_key="<your-anthropic-api-key>"),
) as client:
    async for chunk in client.agenerate(
        model="your-anthropic-model",
        messages=[UserMessage(content="Explain recursion briefly.")],
        stream=True,
    ):
        if chunk.type == "content":
            print(chunk.chunk, end="")

Package Layout

• llmai/openai: OpenAI adapter
• llmai/azure: Azure OpenAI adapter
• llmai/vertex: Vertex AI adapter
• llmai/deepseek: DeepSeek adapter
• llmai/openrouter: OpenRouter adapter
• llmai/cerebras: Cerebras adapter
• llmai/fireworks: Fireworks adapter
• llmai/anthropic: Anthropic adapter
• llmai/google: Google Gemini adapter
• llmai/bedrock: Amazon Bedrock adapter
• llmai/litellm: LiteLLM adapter
• llmai/shared: common message, tool, schema, and response models

Core Types

The shared layer includes the main primitives you will use across providers:

• UserMessage, SystemMessage, AssistantMessage
• TextContentPart, ImageContentPart
• Tool, WebSearchTool, ToolResponseMessage
• JSONSchemaResponse, JSONObjectResponse, TextResponse
• ResponseContent, ResponseStreamChunk, ResponseStreamContentChunk, ResponseStreamThinkingChunk, ResponseStreamToolChunk, ResponseStreamToolCompleteChunk, ResponseStreamCompletionChunk
• AsyncBaseClient, AsyncResponseResult
• ResponseUsage

UserMessage.content accepts either a plain string or explicit content parts. SystemMessage.content is always a plain string.