Latest News

• [2026/09] TensorRT Edge-LLM completes the MLPerf Edge Agentic benchmark 6.4x faster on Jetson AGX Thor.
• [2026/09] Release 0.11.0 adds published Python wheels, Muse-Glimmer, experimental pi0.5, Hunyuan dense models, guided decoding, opt-in in-flight batching, native provider Jinja chat templates, and GPU-free CuTe DSL artifact generation.
• [2026/09] Release 0.10.1 adds experimental TP=2 inference on Dual NVIDIA DGX Spark and redesigns the experimental OpenAI-compatible server for faster cold launches and lower memory usage.
• [2026/08] TensorRT Edge-LLM 0.10.0 adds Day-0 support for Qwen3.8-27B.
• [2026/08] Release 0.10.0 adds support for NVIDIA Nemotron-3.5 Lightning with MTP and DFlash, Cosmos3-Edge, DiffusionGemma, Nemotron-3.5-ASR, and DSpark speculative decoding, alongside an experimental direct TensorRT engine builder without ONNX export, multi-turn KV-cache reuse, and video input for the experimental OpenAI-compatible server.
• [2026/07] Support for the full Gemma 4 family (E2B / E4B / 12B / 26B-A4B / 31B — multimodal text + image + audio, with MTP), Qwen3-Omni and Nemotron-3 NVFP4, and DFlash speculative decoding (with DDTree for Qwen3 / Qwen3.5) landed across releases 0.9.0 and 0.9.1.

---

Overview

TensorRT Edge-LLM is NVIDIA's C++ inference runtime for text, vision, audio, speech, and action models on NVIDIA Jetson, NVIDIA DRIVE, and NVIDIA DGX Spark. The supported frontend exports Hugging Face checkpoints to ONNX for C++ engine building; an experimental direct frontend builds engines from checkpoints without ONNX. Both paths use the same C++ deployment runtimes.

---

Getting Started

Check the Official Support Matrix, then follow the Quick Start Guide. Checkpoint IDs are listed in Supported Models.

For a supported target, install a published Python wheel without compiling Edge-LLM. Use tensorrt-edgellm[server] for the high-level Python API and HTTP serving; see the extras guide for export/tools dependencies and the minimal base workflow.

---

Documentation

Introduction

• Overview - What is TensorRT Edge-LLM and key features
• Official Support Matrix - Platform, JetPack, DriveOS, CUDA, TensorRT, and TensorRT Edge-LLM compatibility
• Supported Models - Complete model compatibility matrix
• Checkpoint Exporter - Recommended ONNX export pipeline
• Experimental Direct Engine Builder - Build all model components directly from a checkpoint

User Guide

• Installation - Set up quantization, tensorrt_edgellm, and the C++ runtime
• Quick Start Guide - Run your first inference in ~15 minutes
• Examples - End-to-end workflows
• Quantization - Create quantized checkpoints for tensorrt_edgellm
• Experimental High-Level Python API and Server - vLLM-style API and OpenAI-compatible server
• Input Format Guide - Request format and specifications
• Chat Template Format - Chat template configuration

Developer Guide

Software Design

• Quantization Package Design - Quantization package architecture
• Engine Builder - Building TensorRT engines
• C++ Runtime Overview - Runtime system architecture
  • LLM Inference Runtime

Advanced Topics

• Customization Guide - Customizing TensorRT Edge-LLM for your needs
• TensorRT Plugins - Custom plugin development
• Tests - Comprehensive test suite for contributors

---

Performance

See the Performance Benchmarks page for released benchmark results covering LLM and VLM prefill, generation throughput, memory usage, and EAGLE speculative decoding speedups.

---

Use Cases

🚗 Automotive

• In-vehicle AI assistants
• Voice-controlled interfaces
• Scene understanding
• Driver assistance systems

🤖 Robotics

• Natural language interaction
• Task planning and reasoning
• Visual question answering
• Human-robot collaboration

🏭 Industrial IoT

• Equipment monitoring with NLP
• Automated inspection
• Predictive maintenance
• Voice-controlled machinery

📱 Edge Devices

• On-device chatbots
• Offline language processing
• Privacy-preserving AI
• Low-latency inference

---

Featured Websites

• TensorRT Edge-LLM Jetson AI Lab tutorial
• Maximizing Memory Efficiency to Run Bigger Models on NVIDIA Jetson
• Build Next-Gen Physical AI with Edge-First LLMs for Autonomous Vehicles and Robotics
• Accelerate AI Inference for Edge and Robotics with NVIDIA Jetson T4000 and NVIDIA JetPack 7.1
• Accelerating LLM and VLM Inference for Automotive and Robotics with NVIDIA TensorRT Edge-LLM

Follow our GitHub repository for the latest updates, releases, and announcements.

---

Support

• Documentation: Full Documentation
• Quick Start: Quick Start Guide
• Roadmap: Developer Roadmap
• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Forums: NVIDIA Developer Forums

---

License

Apache License 2.0

---

Contributing

We welcome contributions! Please see our Contributing Guidelines for details.

---