triton-hf-builder

Turn a downloaded Hugging Face checkpoint into a Triton model repository.

The builder generates Triton configuration, runtime scripts and dependency files for Transformers, vLLM and TensorRT-LLM. Use it through the Python API, the triton-hf CLI, or the code-server extension in Triton Control.

Python API · Serving guide · Serving tests · Contributing

Install

Requires Python 3.10+.

python -m pip install triton-hf-builder

For development from a source checkout, see Contributing.

Quick start

Point source to a complete local checkpoint with weights and tokenizer files. Choose a new output directory outside the checkpoint directory.

Generation copies the checkpoint assets without loading weights and runs without a GPU. Allow disk space for a complete checkpoint copy.

from triton_hf_builder import generate_repository

result = generate_repository(
    source="/workspace/hf-checkpoint",
    output="/workspace/model_repository",
    name="my_model",
    runtime="transformers",
    image="nvcr.io/nvidia/tritonserver:26.06-py3",
    device="cpu",
)
print(result.model_directory / "config.pbtxt")

No manual requirements needed: the builder writes runtime defaults into requirements.txt. Optional overrides, all parameters and return values are explained in the Python API.

The equivalent CLI:

triton-hf inspect /workspace/hf-checkpoint
triton-hf build /workspace/hf-checkpoint \
  --output /workspace/model_repository --name my_model \
  --runtime transformers --device cpu \
  --image nvcr.io/nvidia/tritonserver:26.06-py3

Generated output

For the Transformers example above, a checkpoint with a single Safetensors weight file produces the following layout. Checkpoint filenames vary with the source.

model_repository/
├── my_model/
│   ├── config.pbtxt                 # Triton model configuration
│   └── 1/
│       ├── model.py                 # Triton Python adapter
│       ├── settings.json            # Task and generation settings
│       └── model/
│           ├── config.json
│           ├── model.safetensors    # Copied checkpoint weights
│           ├── tokenizer.json
│           ├── README.md            # Original model card, if supplied
│           └── LICENSE              # Original model license, if supplied
├── requirements.txt                # Serving dependencies
├── serving.json                    # Runtime, image and build metadata
├── LICENSE.triton-hf-builder        # License for generated support scripts
└── README.md                       # Serving instructions

TensorRT-LLM uses the same layout with its own Python adapter. For vLLM, my_model/1/model.json replaces model.py and settings.json, and prepare_repository.py is added at the repository root.

Deploy the model_repository/ root to Triton after preparing its serving dependencies; generation alone does not validate inference. See the serving guide for deployment and vLLM path preparation.

Adapters and compatibility

The API parameter runtime selects the builder adapter. The Triton backend and model execution framework are separate layers.

Recognized multimodal checkpoints and custom model Python are outside the current adapter scope. See runtime limitations.

Manufacturer compatibility catalog

Architecture support reported by the backend projects; individual models may require additional configuration.

Source: NVIDIA image matrix. † Upstream base version; NVIDIA build differs.

Full catalog: architecture names, exact engine versions, CUDA and reported driver versions. See the linked backend documentation for quantization, GPU and feature requirements.

The catalog tracks 26.06 and the latest documented image per runtime. A weekly workflow proposes updates as PRs. Maintenance and selection policy.

Try it with Docker

Run a complete CPU serving example from a repository checkout:

./examples/docker/quickstart.sh

The launcher downloads a small pinned checkpoint, generates its model repository, builds the serving image, waits for readiness and sends an inference request. No GPU or local Python installation is needed. The first run downloads large images and dependencies; later runs reuse cached files and build layers. See the Docker quickstart for requirements, requests and cleanup.

Development and releases

See CONTRIBUTING.md for local checks and RELEASING.md for package publication and code-server integration. Real GPU serving tests are opt-in; follow the test guide.

Apache-2.0 license · Licensing and test models · Security policy · Code of Conduct Model weights and serving engines retain their own licenses.