triton-hf-builder

Turn a downloaded Hugging Face checkpoint into a Triton model repository.

Use the Python library, CLI or the code-server extension in Triton Control. Generation reads metadata and copies assets; it does not load weights or require a GPU. Allow disk space for a full checkpoint copy. Status: alpha.

Python API · Serving guide · Serving tests · Contributing

Install

Requires Python 3.10+.

git clone https://github.com/ai-lab-tech/triton-hf-builder.git
cd triton-hf-builder
python3 -m venv .venv
source .venv/bin/activate
python -m pip install .

Quick start

Point source to a complete local checkpoint with weights and tokenizer files. Choose a new output directory outside the checkpoint directory.

from triton_hf_builder import generate_repository

result = generate_repository(
    source="/workspace/hf-checkpoint",
    output="/workspace/model_repository",
    name="my_model",
    runtime="transformers",
    image="nvcr.io/nvidia/tritonserver:26.06-py3",
    device="cpu",
)
print(result.model_directory / "config.pbtxt")

No manual requirements needed: the builder writes runtime defaults into requirements.txt. Optional overrides, all parameters and return values are explained in the Python API.

The equivalent CLI:

triton-hf inspect /workspace/hf-checkpoint
triton-hf build /workspace/hf-checkpoint \
  --output /workspace/model_repository --name my_model \
  --runtime transformers --device cpu \
  --image nvcr.io/nvidia/tritonserver:26.06-py3

The output contains the checkpoint assets, config.pbtxt, runtime files, requirements.txt and serving.json. Deploy this repository to Triton after preparing its serving dependencies; generation alone does not validate inference. See the serving guide for deployment and vLLM path preparation.

Adapters and compatibility

The API parameter runtime selects the builder adapter. The Triton backend and model execution framework are separate layers.

Recognized multimodal checkpoints and custom model Python are outside the current adapter scope. See runtime limitations.

Manufacturer compatibility catalog

Manufacturer-documented text-generation candidates. Full catalog and architecture names.

Source: NVIDIA image matrix. † Upstream base version; NVIDIA build differs.

These are documentation references, not serving guarantees. Check the linked matrices for quantization, GPU and feature restrictions; the reported driver is not a universal minimum.

The catalog tracks 26.06 and the latest documented image per runtime. A weekly workflow proposes updates as PRs. Maintenance and selection policy.

Development and releases

See CONTRIBUTING.md for local checks and RELEASING.md for package publication and code-server integration. Real GPU serving tests are opt-in; follow the test guide.

Apache-2.0 license · Security policy Model weights and serving engines retain their own licenses.