b10-xgrammar 0.1.23rc2

Efficient, Flexible and Portable Structured Generation

Efficient, Flexible and Portable Structured Generation

Get Started | Documentation | Blogpost | Technical Report

News

• [2025/02] XGrammar has been officially integrated into Modular's MAX
• [2025/01] XGrammar has been officially integrated into TensorRT-LLM.
• [2024/12] XGrammar has been officially integrated into vLLM.
• [2024/12] We presented research talks on XGrammar at CMU, UC Berkeley, MIT, THU, SJTU, Ant Group, LMSys, Qingke AI, Camel AI. The slides can be found here.
• [2024/11] XGrammar has been officially integrated into SGLang.
• [2024/11] XGrammar has been officially integrated into MLC-LLM.
• [2024/11] We officially released XGrammar v0.1.0!

Overview

XGrammar is an open-source library for efficient, flexible, and portable structured generation.

It leverages constrained decoding to ensure 100% structural correctness of the output. It supports general context-free grammar to enable a broad range of structures, including JSON, regex, custom context-free grammar, etc.

XGrammar uses careful optimizations to achieve extremely low overhead in structured generation. It has achieved near-zero overhead in JSON generation, making it one of the fastest structured generation engines available.

XGrammar features universal deployment. It supports:

• Platforms: Linux, macOS, Windows
• Hardware: CPU, NVIDIA GPU, AMD GPU, Apple Silicon, TPU, etc.
• Languages: Python, C++, and JavaScript APIs
• Models: Qwen, Llama, DeepSeek, Phi, Gemma, etc.

XGrammar is very easy to integrate with LLM inference engines. It is the default structured generation backend for most LLM inference engines, including vLLM, SGLang, TensorRT-LLM, and MLC-LLM, as well as many other companies. You can also try out their structured generation modes!

Get Started

Install XGrammar:

pip install xgrammar

Import XGrammar:

import xgrammar as xgr

Please visit our documentation to get started with XGrammar.

• Installation
• Quick start

Adoption

XGrammar has been adopted by many projects and companies, including but not limited to:

            WebLLM

Citation

If you find XGrammar useful in your research, please consider citing our paper:

@article{dong2024xgrammar,
  title={Xgrammar: Flexible and efficient structured generation engine for large language models},
  author={Dong, Yixin and Ruan, Charlie F and Cai, Yaxing and Lai, Ruihang and Xu, Ziyi and Zhao, Yilong and Chen, Tianqi},
  journal={Proceedings of Machine Learning and Systems 7},
  year={2024}
}