langchain-gmi

LangChain integration for GMI Cloud's OpenAI-compatible Inference Engine. Access DeepSeek V3.2, Claude Sonnet 4.5, and open models through one endpoint, alongside GMI Cloud's NVIDIA H100 GPU inference infrastructure. This is a community integration.

Install

pip install langchain-gmi

Set an API key in your environment:

export GMI_API_KEY="your-gmi-api-key"

PowerShell:

$env:GMI_API_KEY = "your-gmi-api-key"

Requires Python 3.10 or newer. The default endpoint is https://api.gmi-serving.com/v1. The integration accepts gmi_api_key, api_key, or openai_api_key; GMI_API_KEY is the environment variable fallback. Explicit credentials take precedence. Missing or empty credentials raise ValueError during construction. OPENAI_API_KEY and OpenAI endpoint environment variables are not used to configure this provider.

llm = ChatGMI(
    model="deepseek-v3.2",
    # gmi_api_key=secret_from_your_secret_manager,
    base_url="https://api.gmi-serving.com/v1",
    timeout=60,
    max_retries=2,
    metadata={"application": "support"},
)

Runs include provider="gmi-cloud" metadata and ls_provider="gmi-cloud" for tracing. API keys use Pydantic secret fields and are excluded from provider-specific exports.

Quickstart

from langchain_gmi import ChatGMI

llm = ChatGMI(model="deepseek-v3.2")
response = llm.invoke("What makes GMI Cloud H100 clusters fast?")
print(response.content)

ChatGMI inherits LangChain's chat model methods, including invoke/ainvoke, stream/astream, bind_tools, and with_structured_output. Availability and behavior of model capabilities depend on the selected model and the API's current support.

Async and streaming

import asyncio

from langchain_gmi import ChatGMI


async def main() -> None:
    llm = ChatGMI(model="deepseek-v3.2")

    response = await llm.ainvoke("Give me a one-sentence summary of inference APIs.")
    print(response.content)

    async for chunk in llm.astream("Count from one to three."):
        print(chunk.content, end="", flush=True)
    print()


if __name__ == "__main__":
    asyncio.run(main())

For synchronous streaming:

for chunk in llm.stream("Count from one to three."):
    print(chunk.content, end="", flush=True)

If your selected model supports usage reporting, pass stream_options={"include_usage": True} to stream or astream and inspect the final chunk's usage_metadata. Streaming usage requests are opt-in for this endpoint. Recent langchain-openai versions also accept stream_usage=True.

Tool calling

Bind LangChain tools to the model, then invoke it with a message. The model may return tool calls in AIMessage.tool_calls; your application is responsible for executing them and returning the results.

from langchain_core.tools import tool

from langchain_gmi import ChatGMI


@tool
def multiply(a: int, b: int) -> int:
    """Multiply two integers."""
    return a * b


llm = ChatGMI(model="deepseek-v3.2").bind_tools([multiply])
message = llm.invoke("What is 6 times 7?")
print(message.tool_calls)

See examples/02_agent_with_tools.py for a bounded tool loop with calculator and web search tools.

Structured output

Use a Pydantic model to validate extracted fields:

from pydantic import BaseModel, Field

from langchain_gmi import ChatGMI


class SupportTicket(BaseModel):
    """Fields extracted from a short support request."""

    category: str = Field(description="Short issue category")
    urgency: int = Field(ge=1, le=5, description="Urgency from 1 (low) to 5 (high)")
    summary: str


extract = ChatGMI(model="deepseek-v3.2").with_structured_output(SupportTicket)
ticket = extract.invoke("My account is locked and our team is blocked from a deadline.")
print(ticket)

The integration's default structured-output strategy is function_calling. See examples/03_structured_output.py.

Models and discovery

The short names below are convenience aliases. You can also pass a raw model ID from GMI Cloud's model catalog; other strings pass through unchanged.

ChatGMI model	GMI model ID	Upstream context reference
deepseek-v3.2	gmi/deepseek-ai/DeepSeek-V3.2	163,840 configured positions
claude-sonnet-4.5	gmi/anthropic/claude-sonnet-4.5	200K tokens
claude-opus-4.5	gmi/anthropic/claude-opus-4.5	200K tokens
qwen3-vl	gmi/Qwen/Qwen3-VL-235B-A22B-Instruct-FP8	256K tokens, native
gpt-5	gmi/openai/gpt-5	400,000 tokens
gemini-3-flash	gmi/google/gemini-3-flash-preview	1,048,576 input tokens

These are upstream references, not verified limits for GMI deployments. The DeepSeek value is its model configuration's position limit. Check the GMI catalog for deployment limits and current availability. The aliases above are fixed convenience mappings; use a raw ID from discovery if your account's catalog uses a different ID.

list_gmi_models() returns each API model record with a normalized context_length integer or None when the API does not report a value. It recognizes positive integer context_length, max_model_len, max_context_length, and context_window fields in that order. It preserves other API metadata and does not infer missing context lengths from this table.

from langchain_gmi import list_gmi_models

for model in list_gmi_models():
    print(model["id"], model["context_length"])

You can also call list_gmi_models(api_key, base_url=..., timeout=30.0, client=httpx_client). A supplied HTTPX client stays open; internally created clients close after the request. HTTP errors raise httpx.HTTPStatusError, timeouts propagate, and malformed model-list responses raise ValueError. Discovery makes one request without automatic retries.

See examples/01_basic_chat.py for chat, streaming, and discovery. GMI's API reference documents GET /v1/models and POST /v1/chat/completions.

Live validation

The integration was tested against GMI Cloud on November 14, 2026 with the catalog model deepseek-ai/deepseek-v4.1-flash-1017:

llm = ChatGMI(model="deepseek-ai/deepseek-v4.1-flash-1017")

Model discovery, sync/async chat, sync/async streaming, tool calling, and Pydantic structured output passed. See the validation record for scope and dependency versions. These are functional checks, not performance benchmarks.

How it fits together

flowchart LR
    A[Your application] --> B[LangChain ChatGMI]
    B -->|OpenAI-compatible chat request| C[GMI Cloud API]
    C --> D[Selected model]
    D -->|chat response or streamed chunks| C
    C --> B
    B --> A
    E[list_gmi_models] -->|GET /v1/models| C

Development

From a checkout of this repository:

pip install -e ".[dev]"
pytest
ruff check .
ruff format --check .
python -m build
twine check dist/*

Tests use mocked HTTP transports, with live HTTP blocked. CI checks Python 3.10 through 3.13 and a minimum LangChain dependency configuration. The MIT license is in LICENSE.

Links

• GMI Cloud developers
• GMI Cloud inference API reference
• LangChain chat models
• GMI Cloud console