LangChain NVIDIA AI Endpoints

The langchain-nvidia-ai-endpoints package contains LangChain integrations for chat models and embeddings powered by NVIDIA AI Foundation Models, and hosted on the NVIDIA API Catalog.

A strong starting point is Nemotron, NVIDIA's open model family built for agentic AI. Nemotron models use a hybrid Mamba-Transformer mixture-of-experts architecture that delivers leading benchmark accuracy with high throughput and up to 1M token context windows. Model weights, training data, and implementation recipes are published openly under the NVIDIA Open Model License.

NVIDIA AI Foundation models are community- and NVIDIA-built models that are optimized to deliver the best performance on NVIDIA-accelerated infrastructure.  You can use the API to query live endpoints that are available on the NVIDIA API Catalog to get quick results from a DGX-hosted cloud compute environment. or you can download models from NVIDIA's API catalog with NVIDIA NIM, which is included with the NVIDIA AI Enterprise license. The ability to run models on-premises gives your enterprise ownership of your customizations and full control of your IP and AI application.

NIM microservices are packaged as container images on a per model/model family basis and are distributed as NGC container images through the NVIDIA NGC Catalog. At their core, NIM microservices are containers that provide interactive APIs for running inference on an AI Model. 

Use this documentation to learn how to install the langchain-nvidia-ai-endpoints package and use it for some common functionality for text-generative and embedding models.

Get Started

Install langchain-nvidia-ai-endpoints

To install the langchain-nvidia-ai-endpoints package, use the following code.

%pip install -U --quiet langchain-nvidia-ai-endpoints

Get Access to the NVIDIA API Catalog

To get access to the NVIDIA API Catalog, do the following:

1. Create a free account on the NVIDIA API Catalog and log in.

2. Click your profile icon, and then click API Keys. The API Keys page appears.

3. Click Generate API Key. The Generate API Key window appears.

4. Click Generate Key. You should see API Key Granted, and your key appears.

5. Copy and save the key as NVIDIA_API_KEY.

6. To verify your key, use the following code.

   import getpass
   import os
   
   if not os.environ.get("NVIDIA_API_KEY", "").startswith("nvapi-"):
       nvidia_api_key = getpass.getpass("Enter your NVIDIA API key: ")
       assert nvidia_api_key.startswith("nvapi-"), f"{nvidia_api_key[:5]}... is not a valid key"
       os.environ["NVIDIA_API_KEY"] = nvidia_api_key
   

You can now use your key to access endpoints on the NVIDIA API Catalog.

Choose a deployment and check it before inference

Hosted NVIDIA API Catalog calls default to https://integrate.api.nvidia.com/v1 and require NVIDIA_API_KEY. Set it in your environment (do not put it in source code or a URL). A self-hosted NIM can use its deployment root (http://localhost:8000) or its OpenAI-compatible API base (http://localhost:8000/v1); both resolve to /v1. Set NVIDIA_BASE_URL or pass base_url= to ChatNVIDIA, NVIDIAEmbeddings, NVIDIA (completions), or NVIDIARerank. An explicit constructor base_url takes precedence over NVIDIA_BASE_URL. Custom reverse-proxy prefixes are retained (for example, https://proxy.example/nim becomes https://proxy.example/nim/v1). Do not pass /models, /chat/completions, /embeddings, /completions, /ranking, /rankings, or /reranking as the base URL: those are endpoint paths, not deployment roots. The connector uses GET <base>/models for discovery and appends /chat/completions, /embeddings, /completions, or /ranking for the corresponding client. Self-hosted deployments may accept anonymous access, but set NVIDIA_API_KEY if yours requires bearer auth. Hosted catalog models with dedicated endpoints may not appear in /models; their connector routing depends on the known hosted-model registry.

Run the read-only Doctor preflight before sending prompts:

python -m langchain_nvidia_ai_endpoints doctor --model nvidia/nemotron-3-super-120b-a12b --capability chat
# Self-hosted NIM; no API key unless your deployment requires one
python -m langchain_nvidia_ai_endpoints doctor --base-url http://localhost:8000/v1 --model my-model --capability embeddings

Doctor uses NVIDIA_BASE_URL (or the hosted default), NVIDIA_API_KEY, and GET /v1/models; it sends no inference request or usage telemetry. --capability accepts chat, embeddings, completions, or ranking. It checks /models model type metadata where available; unknown type is reported as UNKNOWN, not verified by a generated response. It exits 0 when connectivity/auth/model listing checks pass and nonzero for missing hosted auth, rejected keys, bad URLs, unavailable /models, missing models, incompatible declared model types, network failures, or timeouts. Check --help for options; use --timeout 10 for a slow deployment. A preflight pass is not proof that inference succeeds or that a framework is BCB-compatible.

Optional BCB compatibility evidence is available from NIM OSS Manager's authenticated internal-alpha API (not a public anonymous service). It is never queried by default. If your team has approved, scoped read-only Manager consumer access, set NIM_OSS_MANAGER_TOKEN in the environment and opt in explicitly:

python -m langchain_nvidia_ai_endpoints doctor --model nvidia/nemotron-3-super-120b-a12b --capability chat --manager-url https://your-manager-host

Doctor requests the Manager's read-only bcb-public-v1alpha1 compatibility summary for langchain-nvidia and that model. It does not transmit your NVIDIA API key to Manager. Missing access, unknown responses, and stale evidence are labeled unavailable, unknown, or stale, never invented as compatible. An unavailable optional Manager lookup does not turn a working endpoint into a failed connectivity check. Do not put Manager tokens in command-line arguments or URLs.

For a maintainer-reviewed, framework-level compatibility snapshot, see NIM compatibility evidence. The optional BCB review artifact separates verified framework status from unknown individual capabilities; it is not a live release approval or a public API.

For hosted-versus-local setup, streaming/tools, embeddings retrieval and migration from generic OpenAI-compatible clients, see the adoption guide, including its scoped hosted recipe smoke results. Capabilities and compatibility still require deployment-specific evidence.

For the documented gaps in OCR, page-elements, table-structure, and offline ASR support (API shapes, proposed abstractions, and future validation cases), see the non-text NIM modality guidance. This is a maintainer proposal, not implemented modality support.

Connector maintainers: the reliability queue groups open issues by root cause, evidence, owner role, and next release or backlog decision; pending fixes are not claimed as released.

Invoke the Core Chat Interface

Use the following code to invoke the core chat interface.

## Core LC Chat Interface
from langchain_nvidia_ai_endpoints import ChatNVIDIA

llm = ChatNVIDIA(model="nvidia/nemotron-3-super-120b-a12b")
result = llm.invoke("Write a ballad about LangChain.")
print(result.content)

Use Stream, Batch, and Async

The models exposed by the NVIDIA API natively support streaming, and they expose a batch method to handle concurrent requests, as well as async methods for invoke, stream, and batch.

The following examples demonstrate how to use batch and stream, and their async versions.

# Batch example
print(llm.batch(["What's 2*3?", "What's 2*6?"]))

# Batch example (async)
await llm.abatch(["What's 2*3?", "What's 2*6?"])

# Stream example
for chunk in llm.stream("How far can a seagull fly in one day?"):
    # Show the token separations
    print(chunk.content, end="|")

# Stream example (async)
async for chunk in llm.astream("How long does it take for monarch butterflies to migrate?"):
    print(chunk.content, end="|")

Get a List of Supported Models

You can query available_models to get a list of the models that you can access with your API credentials. Use the following code.

[model.id for model in llm.available_models if model.model_type]

You should see output similar to the following.

#[
# ...
# 'databricks/dbrx-instruct',
# 'google/codegemma-7b',
# 'google/gemma-2b',
# 'google/gemma-7b',
# 'google/recurrentgemma-2b',
# 'meta/codellama-70b',
# 'meta/llama2-70b',
# 'meta/llama3-70b-instruct',
# 'meta/llama3-8b-instruct',
# 'microsoft/phi-3-mini-128k-instruct',
# 'mistralai/mistral-7b-instruct-v0.2',
# 'mistralai/mistral-large',
# 'mistralai/mixtral-8x22b-instruct-v0.1',
# 'mistralai/mixtral-8x7b-instruct-v0.1',
# 'snowflake/arctic',
# ...
#]

Work With Different Model Types

Some model types support unique prompting techniques and chat messages. Use this section to learn about a few examples.

Nemotron

Nemotron is NVIDIA's open model family optimized for agentic AI. The models deliver strong reasoning and tool-calling performance with high throughput. nvidia/nemotron-3-super-120b-a12b is a recommended starting point: a powerful model with 12B active parameters and 120B parameters in total, with a 1M token context window, available on the NVIDIA API Catalog and as a self-hosted NIM.

from langchain_nvidia_ai_endpoints import ChatNVIDIA

llm = ChatNVIDIA(model="nvidia/nemotron-3-super-120b-a12b")
result = llm.invoke("Plan a three-step agentic workflow for competitive research.")
print(result.content)

Nemotron 3.5 Super VL

Nemotron 3.5 Super VL accepts text and images through ChatNVIDIA.

The connector defaults this model to 8,192 completion tokens. Set a thinking preference as needed; pass max_completion_tokens to override the default:

from langchain_nvidia_ai_endpoints import ChatNVIDIA

super_model = ChatNVIDIA(
    model="nvidia/nemotron-3.5-super-vl-120b-a12b",
    chat_template_kwargs={"enable_thinking": True},
)
result = super_model.invoke("Plan a three-step agentic workflow for competitive research.")
print(result.content_blocks)

To describe an image, replace photo.png with a local PNG and pass a standard image content block:

import base64
from pathlib import Path

from langchain_core.messages import HumanMessage

image = base64.b64encode(Path("photo.png").read_bytes()).decode("ascii")
response = super_model.invoke([
    HumanMessage(content=[
        {"type": "text", "text": "Describe this image."},
        {"type": "image", "base64": image, "mime_type": "image/png"},
    ])
])
print(response.content_blocks)

General Chat

Models such as nvidia/nemotron-3-super-120b-a12b and mistralai/mixtral-8x22b-instruct-v0.1 are good all-around models that you can use for any LangChain chat messages.

The following example generates a simple chat response.

from langchain_nvidia_ai_endpoints import ChatNVIDIA
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are a helpful AI assistant named Fred."),
        ("user", "{input}")
    ]
)
chain = (
    prompt
    | ChatNVIDIA(model="nvidia/nemotron-3-super-120b-a12b")
    | StrOutputParser()
)

for txt in chain.stream({"input": "What's your name?"}):
    print(txt, end="")

Code Generation

Code generation models, such as meta/codellama-70b and google/codegemma-7b, tend to perform better on code-generation and structured code tasks.

The following example generates python code to solve a problem.

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are an expert coding AI. Respond only in valid python; no narration whatsoever."),
        ("user", "{input}")
    ]
)
chain = (
    prompt
    | ChatNVIDIA(model="meta/codellama-70b", max_completion_tokens=419)
    | StrOutputParser()
)

for txt in chain.stream({"input": "How do I solve this fizz buzz problem?"}):
    print(txt, end="")

Multimodal Support

NVIDIA also has models that support multimodal inputs, such as nvidia/neva-22b. You can provide both images and text for the model to reason over. These models accept LangChain's standard image formats. Local file paths are no longer accepted directly. If you are upgrading code that passed local paths, read trusted local files explicitly and pass them as data:image/...;base64,... URIs.

The following example asks the model to describe an image.

import requests
import base64

# Initialize the image content
image_url = "https://picsum.photos/seed/kitten/300/200"
image_content = requests.get(image_url).content

# Initialize the model
from langchain_nvidia_ai_endpoints import ChatNVIDIA
llm = ChatNVIDIA(model="nvidia/neva-22b")

# Pass an image as a URL
from langchain_core.messages import HumanMessage

llm.invoke(
    [
        HumanMessage(content=[
            {"type": "text", "text": "Describe this image:"},
            {"type": "image_url", "image_url": {"url": image_url}},
        ])
    ])

# Pass an image as a base64 encoded string
b64_string = base64.b64encode(image_content).decode('utf-8')
llm.invoke(
    [
        HumanMessage(content=[
            {"type": "text", "text": "Describe this image:"},
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64_string}"}},
        ])
    ])

The NVIDIA API uniquely accepts images as base64 images within HTML img tags. While this isn't interoperable with other LLMs, you can prompt the model accordingly as shown in the following example.

base64_with_mime_type = f"data:image/png;base64,{b64_string}"
llm.invoke(
    f"What's in this image?\n<img src=\"{base64_with_mime_type}\" />"
)

Completions

You can work with models that support the Completions API. These models accept a prompt instead of messages.

The following example gets a list of models that support the Completions API.

completions_llm = NVIDIA().bind(max_tokens=512)
[model.id for model in completions_llm.get_available_models()]

# [
#   ...
#   'bigcode/starcoder2-7b',
#   'bigcode/starcoder2-15b',
#   ...
# ]

The following example uses the Completions API to generate a code example.

prompt = "# Function that does quicksort written in Rust without comments:"
for chunk in completions_llm.stream(prompt):
    print(chunk, end="", flush=True)

Dynamo KV Cache Optimization

ChatNVIDIADynamo is a drop-in replacement for ChatNVIDIA that injects NVIDIA Dynamo KV cache routing hints into requests. Use it with Dynamo-enabled deployments to improve inference scheduling.

from langchain_nvidia_ai_endpoints import ChatNVIDIADynamo

llm = ChatNVIDIADynamo(model="nvidia/nemotron-3-super-120b-a12b", osl=1024, iat=100)
result = llm.invoke("Write a short poem.")

Embeddings

The following example connects to an embeddings model.

from langchain_nvidia_ai_endpoints import NVIDIAEmbeddings

embedder = NVIDIAEmbeddings(model="NV-Embed-QA")
embedder.embed_query("What's the temperature today?")
embedder.embed_documents([
    "The temperature is 42 degrees.",
    "Class is dismissed at 9 PM."
])

Ranking

The following example connects to a ranking model.

from langchain_nvidia_ai_endpoints import NVIDIARerank
from langchain_core.documents import Document

query = "What is the GPU memory bandwidth of H100 SXM?"
passages = [
    "The Hopper GPU is paired with the Grace CPU using NVIDIA's ultra-fast chip-to-chip interconnect, delivering 900GB/s of bandwidth, 7X faster than PCIe Gen5. This innovative design will deliver up to 30X higher aggregate system memory bandwidth to the GPU compared to today's fastest servers and up to 10X higher performance for applications running terabytes of data.",
    "A100 provides up to 20X higher performance over the prior generation and can be partitioned into seven GPU instances to dynamically adjust to shifting demands. The A100 80GB debuts the world's fastest memory bandwidth at over 2 terabytes per second (TB/s) to run the largest models and datasets.",
    "Accelerated servers with H100 deliver the compute power—along with 3 terabytes per second (TB/s) of memory bandwidth per GPU and scalability with NVLink and NVSwitch™.",
]

client = NVIDIARerank(model="nvidia/llama-3.2-nv-rerankqa-1b-v1")

response = client.compress_documents(
  query=query,
  documents=[Document(page_content=passage) for passage in passages]
)

print(f"Most relevant: {response[0].page_content}\nLeast relevant: {response[-1].page_content}")

Reranked results are separate Document objects with metadata["relevance_score"]. The input documents and earlier result scores remain unchanged across subsequent sync or async calls, so callers can safely reuse the same source documents.

Ranking with a Vision-Language Rerank Model

NVIDIARerank can also rerank documents that carry an image alongside their text using a ranking VLM such as nvidia/llama-nemotron-rerank-vl-1b-v2.

The following example connects to a ranking VLM.

from langchain_nvidia_ai_endpoints import NVIDIARerank
from langchain_core.documents import Document

query = "Show me a picture of a cat"
image = "data:image/png;base64,<base64-encoded-image>"
documents = [
    Document(
        page_content="The picture of a cat.",
        metadata={"image": image},
    ),
    Document(
        page_content="",
        metadata={"image": image},
    ),
    Document(page_content="The weather today is sunny."),
]

ranker = NVIDIARerank(model="nvidia/llama-nemotron-rerank-vl-1b-v2")
response = ranker.compress_documents(query=query, documents=documents)

Retrieval

The following example connects to a running NVIDIA RAG Blueprint server to retrieve relevant documents via the /v1/search endpoint.

from langchain_nvidia_ai_endpoints import NVIDIARetriever

retriever = NVIDIARetriever(base_url="http://localhost:8081", k=4)
docs = retriever.invoke("What is machine learning?")

Self-host with NVIDIA NIM Microservices

When you are ready to deploy your AI application, you can self-host models with NVIDIA NIM. For more information, refer to NVIDIA NIM Microservices.

The following code connects to locally hosted NIM Microservices.

from langchain_nvidia_ai_endpoints import ChatNVIDIA, NVIDIAEmbeddings, NVIDIARerank

# Connect to an chat NIM running at localhost:8000, and specify a model
llm = ChatNVIDIA(base_url="http://localhost:8000/v1", model="nvidia/nemotron-3-super-120b-a12b")

# Connect to an embedding NIM running at localhost:8080
embedder = NVIDIAEmbeddings(base_url="http://localhost:8080/v1")

# Connect to a reranking NIM running at localhost:2016
ranker = NVIDIARerank(base_url="http://localhost:2016/v1")

If your deployment exposes a direct inference URL instead of a /v1/models listing plus a standard inference path, register it as a model endpoint rather than passing the inference URL as base_url. A 404 from base_url plus /embeddings usually means the deployment URL is already the full inference endpoint.

import os

from langchain_nvidia_ai_endpoints import Model, NVIDIAEmbeddings, register_model

register_model(
    Model(
        id="nvidia/llama-3.2-nv-embedqa-1b-v2",
        model_type="embedding",
        client="NVIDIAEmbeddings",
        endpoint="https://CUSTOM_DEPLOYMENT_LINK_WITHOUT_TRAILING_EMBEDDINGS",
    )
)

embedder = NVIDIAEmbeddings(
    model="nvidia/llama-3.2-nv-embedqa-1b-v2",
    api_key=os.environ.get("NVIDIA_API_KEY"),
)

Telemetry & Privacy

langchain-nvidia-ai-endpoints includes content-free, aggregate telemetry to help NVIDIA understand adoption and reliability of supported NVIDIA NIM integrations. Telemetry is enabled by default for supported NVIDIA-hosted NIM endpoint usage and can be disabled at any time.

Telemetry is limited to aggregate operational metrics such as connector and framework version, operation type, recognized model family, request and outcome counts, token counts, latency buckets, and coarse error categories. These totals are best-effort and are not billing-grade.

The telemetry is aggregate-only and does not include request or response content, credentials, endpoint URLs, or persistent user, device, or session identifiers. It is not used to track individual users, for billing, or for precise capacity accounting.

Telemetry is aggregated in memory and emitted only for supported NVIDIA-hosted NIM endpoints. Self-hosted endpoints do not emit this telemetry.

You may opt out of telemetry collection at any time. Opting out applies only to telemetry collection by the langchain-nvidia-ai-endpoints connector itself. To disable telemetry for the current shell, set NVIDIA_USAGE_TELEMETRY_ENABLED=false before running your application:

export NVIDIA_USAGE_TELEMETRY_ENABLED=false

You can also disable telemetry for a single client instance:

llm = ChatNVIDIA(
    model="nvidia/nemotron-3-super-120b-a12b",
    usage_telemetry_enabled=False,
)

If telemetry is disabled, the connector does not send telemetry requests and telemetry has no effect on connector operation.

Use of third-party endpoints, including NVIDIA Build: langchain-nvidia-ai-endpoints can be configured to use various inference endpoints, including build.nvidia.com (NVIDIA Build). If you choose to use NVIDIA Build or any other third-party endpoint, that endpoint's own terms of service and privacy practices apply independently of this library. Any opt-out you exercise within langchain-nvidia-ai-endpoints does not extend to data collection by your chosen endpoint. NVIDIA Build is intended for evaluation and testing purposes only and may not be used in production environments. Do not submit confidential information or personal data when using NVIDIA Build.