ai-dynamo-llm

OpenAI-compatible inference client with multimodal support, designed for NVIDIA Dynamo-style serving stacks.

Features

• Chat completions (streaming and non-streaming) against any OpenAI-compatible endpoint
• Multimodal messages: attach images by file path or URL (base64 data-URI encoding)
• Text embeddings
• Automatic retry with backoff on 429 / 5xx / connection errors
• Token estimation with CJK-aware budgeting

Install

pip install ai-dynamo-llm

Quick start

from ai_dynamo_llm import AIDynamoClient, ChatMessage

client = AIDynamoClient(api_key="sk-...", base_url="http://localhost:8000/v1")

resp = client.chat(
    model="dynamo-llm",
    messages=[
        ChatMessage.system("You are a helpful assistant."),
        ChatMessage.user("What is Dynamo?"),
    ],
)
print(resp["choices"][0]["message"]["content"])

Multimodal

msg = ChatMessage.multimodal(
    "Describe this image.",
    ["/tmp/screenshot.png", "https://example.com/diagram.png"],
)
resp = client.chat(model="dynamo-vlm", messages=[msg])

Token budgeting

from ai_dynamo_llm import estimate_tokens, truncate_to_tokens

assert estimate_tokens("hello world") <= estimate_tokens("你好世界" * 10)
prompt = truncate_to_tokens(long_doc, max_tokens=2048)

License

Apache-2.0