Latest News

• [2026/09] TensorRT Edge-LLM completes the MLPerf Edge Agentic benchmark 6.4x faster on Jetson AGX Thor.
• [2026/09] Release 0.11.0 adds published Python wheels, Muse-Glimmer, experimental pi0.5, Hunyuan dense models, guided decoding, opt-in in-flight batching, native provider Jinja chat templates, and GPU-free CuTe DSL artifact generation.
• [2026/09] Release 0.10.1 adds experimental TP=2 inference on Dual NVIDIA DGX Spark and redesigns the experimental OpenAI-compatible server for faster cold launches and lower memory usage.
• [2026/08] TensorRT Edge-LLM 0.10.0 adds Day-0 support for Qwen3.8-27B.
• [2026/08] Release 0.10.0 adds support for NVIDIA Nemotron-3.5 Lightning with MTP and DFlash, Cosmos3-Edge, DiffusionGemma, Nemotron-3.5-ASR, and DSpark speculative decoding, alongside an experimental direct TensorRT engine builder without ONNX export, multi-turn KV-cache reuse, and video input for the experimental OpenAI-compatible server.
• [2026/07] Support for the full Gemma 4 family (E2B / E4B / 12B / 26B-A4B / 31B — multimodal text + image + audio, with MTP), Qwen3-Omni and Nemotron-3 NVFP4, and DFlash speculative decoding (with DDTree for Qwen3 / Qwen3.5) landed across releases 0.9.0 and 0.9.1.

---

Overview

TensorRT Edge-LLM is NVIDIA's C++ inference runtime for text, vision, audio, speech, and action models on NVIDIA Jetson, NVIDIA DRIVE, and NVIDIA DGX Spark. The supported frontend exports Hugging Face checkpoints to ONNX for C++ engine building; an experimental direct frontend builds engines from checkpoints without ONNX. Both paths use the same C++ deployment runtimes.

---

Getting Started

For a supported target, install a published Python wheel without compiling Edge-LLM. Use tensorrt-edgellm[runtime] for the high-level Python API and HTTP serving. See Optional Python dependencies for additional tools, or Install without optional dependencies for the base installation.

Follow the Quick Start Guide to start a chat or send an inference request. Checkpoint IDs are listed in Supported Models.

Install from PyPI (requires a supported CUDA/TensorRT stack):

python -m pip install "tensorrt-edgellm[runtime]"

---

Use with coding agents

The repository ships agent skills in skills/. Agent hosts that support .agents/skills (including Codex and Cursor) discover them inside a clone; Claude Code uses the .claude/skills link. Set up the project and let the agent drive:

git clone https://github.com/NVIDIA/TensorRT-Edge-LLM.git && cd TensorRT-Edge-LLM
claude "Set up TensorRT Edge-LLM on this machine with the edge-llm-install skill. Ask before system-wide changes."   # or: codex "..."

To use the skills outside a clone, ask: Install the Edge-LLM agent skills for <Claude Code|Codex> with the edge-llm-install skill.

Each skill states the inputs it needs and the evidence it reports before it declares success.

Task	Example prompt	Skill
Install, update, or remove	Install Edge-LLM from a wheel on this Jetson Thor and verify it.	edge-llm-install
Serve a model	Serve <org/model> through the OpenAI-compatible API, send a streaming request, and stop it after the smoke test.	edge-llm-serve
Convert and run a model	Export <org/model>, build the required engines, run C++ inference, and compare with the provider reference.	edge-llm-convert-model
Add or customize a model	Add support for <org/model>; start from its config.json and checkpoint tensors.	edge-llm-add-model
CuTe DSL kernel	Add a CuTe DSL kernel variant for sm_<NN> and validate it on freshly generated artifacts.	edge-llm-cutedsl-kernel
TensorRT plugin	Add a TensorRT plugin for <op> to both frontends.	edge-llm-plugin-extension
Upgrade or backport	Migrate my deployment from Edge-LLM <old> to <new>.	edge-llm-version-migration
Accuracy mismatch	Output differs from the Hugging Face reference; find the first failing layer.	edge-llm-few-layer-test, then edge-llm-accuracy-debugging
Run or add tests	Add an end-to-end case for <org/model> and run only that case.	edge-llm-e2e-test, edge-llm-cpp-test

Not sure which skill applies? Ask the agent to list the skills in skills/ and pick one.

---

Documentation

Introduction

• Overview - What is TensorRT Edge-LLM and key features
• Official Support Matrix - Platform, JetPack, DriveOS, CUDA, TensorRT, and TensorRT Edge-LLM compatibility
• Supported Models - Complete model compatibility matrix
• Checkpoint Exporter - Recommended ONNX export pipeline
• Experimental Direct Engine Builder - Build all model components directly from a checkpoint

User Guide

• Installation - Set up quantization, tensorrt_edgellm, and the C++ runtime
• trtedge CLI - Chat, serve models, build engines, export ONNX, and prepare checkpoints
• Quick Start Guide - Run your first inference in ~15 minutes
• Examples - End-to-end workflows
• Quantization - Create quantized checkpoints for tensorrt_edgellm
• Python API and OpenAI-Compatible Server - vLLM-style API and OpenAI-compatible server
• Input Format Guide - Request format and specifications
• Chat Template Format - Chat template configuration

Developer Guide

Software Design

• Quantization Package Design - Quantization package architecture
• Engine Builder - Building TensorRT engines
• C++ Runtime Overview - Runtime system architecture
  • LLM Inference Runtime

Advanced Topics

• Customization Guide - Customizing TensorRT Edge-LLM for your needs
• TensorRT Plugins - Custom plugin development
• Tests - Comprehensive test suite for contributors

---

Performance

See the Performance Benchmarks page for released benchmark results covering LLM and VLM prefill, generation throughput, memory usage, and EAGLE speculative decoding speedups.

---

Use Cases

🚗 Automotive

• In-vehicle AI assistants
• Voice-controlled interfaces
• Scene understanding
• Driver assistance systems

🤖 Robotics

• Natural language interaction
• Task planning and reasoning
• Visual question answering
• Human-robot collaboration

🏭 Industrial IoT

• Equipment monitoring with NLP
• Automated inspection
• Predictive maintenance
• Voice-controlled machinery

📱 Edge Devices

• On-device chatbots
• Offline language processing
• Privacy-preserving AI
• Low-latency inference

---

Featured Websites

• TensorRT Edge-LLM Jetson AI Lab tutorial
• Run AI Models for Edge Application Development with Just a Few Commands
• Maximizing Memory Efficiency to Run Bigger Models on NVIDIA Jetson
• Build Next-Gen Physical AI with Edge-First LLMs for Autonomous Vehicles and Robotics
• Accelerate AI Inference for Edge and Robotics with NVIDIA Jetson T4000 and NVIDIA JetPack 7.1
• Accelerating LLM and VLM Inference for Automotive and Robotics with NVIDIA TensorRT Edge-LLM

Follow our GitHub repository for the latest updates, releases, and announcements.

---

Support

• Documentation: Full Documentation
• Quick Start: Quick Start Guide
• Roadmap: Developer Roadmap
• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Forums: NVIDIA Developer Forums

---

License

Apache License 2.0

---

Contributing

We welcome contributions! Please see our Contributing Guidelines for details.

---