hf-serve

Installation

First you need to setup your environment with uv (or with your preferred Python environment manager).

uv venv --python 3.12
source .venv/bin/activate

Install it from the uv.lock file for CPU / MPS as follows:

uv sync --active --frozen --extra cpu

Alternatively, install it on NVIDIA CUDA 12.6 as follows:

uv sync --active --frozen --extra cuda --extra flash-attn --preview-features extra-build-dependencies

Or if you're on CUDA 12.8 then:

uv sync --active --frozen --extra cuda-128 --extra flash-attn --preview-features extra-build-dependencies

Or for CUDA 13.0, including NVIDIA GB10 / DGX Spark:

uv sync --active --frozen --extra cuda-130

$ uv run hf-serve --help

KenLM is optional because its installation requires compiling a C++ extension. EmbeddingGemma 2 and normal audio inference do not need it. For ASR models that use KenLM language-model decoding, add --extra kenlm alongside your backend:

uv sync --active --frozen --extra cpu --extra kenlm

Video inputs require TorchCodec and system FFmpeg libraries. TorchCodec is included in the dependencies; this checkout selects its CPU wheels for all model backends, so video decoding does not require a separate CUDA runtime. If updating an existing environment, run uv sync --active --frozen --extra cpu (or your CUDA backend extra).

Container images

On each push to main, CI publishes images to ghcr.io/<owner>/hf-serve using the first seven characters of the commit SHA as the tag. Images are built on native runners for the architectures listed below:

Tag	Backend	Architectures
<sha>	CUDA 12.6	linux/amd64, linux/arm64
<sha>-cu130-arm64	CUDA 13.0, tested on NVIDIA GB10 / DGX Spark	linux/arm64
<sha>-cpu	CPU	linux/amd64, linux/arm64

For example, to pull the CUDA 13.0 image published by the upstream repository:

docker pull ghcr.io/huggingface/hf-serve:<sha>-cu130-arm64

Use a commit published after multi-platform support was added; older tags may only contain AMD64 images. Docker selects the architecture for the host.

To build the CUDA 13.0 variant locally from this checkout:

docker buildx build --platform linux/arm64 \
  --build-arg BASE_IMAGE=nvidia/cuda:13.0.2-devel-ubuntu24.04 \
  --build-arg TORCH_EXTRA=cuda-130 \
  --tag hf-serve:gb10 --load .

Examples

🤏 Run HuggingFaceTB/SmolLM3-3B with an OpenAI API

uv run hf-serve --model-id HuggingFaceTB/SmolLM3-3B --task text-generation --dtype float16

📏 Run google/embeddinggemma-2

EmbeddingGemma 2 supports text, images, audio, video, and interleaved inputs through Sentence Transformers. It requires Transformers 5.19+ and Sentence Transformers 6.1+.

uv run hf-serve --model-id google/embeddinggemma-2 --task feature-extraction

For embedding tasks, --device auto selects a single available device rather than sharding the model across GPUs. To select a specific GPU on a multi-GPU host:

CUDA_VISIBLE_DEVICES=0 uv run hf-serve --model-id google/embeddinggemma-2 --task feature-extraction --device cuda

The default precision is bfloat16 on CUDA devices that support it and float32 elsewhere. Use --dtype float32 or --dtype bfloat16 to override it; float16 is rejected because it produces invalid embeddings for this model.

Choose SearchQuery for search queries and Document for untitled documents. Other prompt names, including CodeRetrieval and SentenceSimilarity, are listed in the model card. No task prefix is added unless a prompt is selected.

curl http://localhost:8080/predict -H 'Content-Type: application/json' \
  -d '{"inputs":"What causes the northern lights?","prompt_name":"SearchQuery","dimensions":256}'

curl http://localhost:8080/v1/embeddings -H 'Content-Type: application/json' \
  -d '{"model":"google/embeddinggemma-2","input":"The northern lights are caused by charged particles from the sun.","prompt_name":"Document","dimensions":256}'

Both routes accept a single modality dictionary or a batch of dictionaries as well as the existing text strings. Media values can be URLs or server-local paths. Images also support base64. A list of media values inside one dictionary produces one shared embedding. You should can use the model's placeholder tokens to position media within text:

curl http://localhost:8080/predict -H 'Content-Type: application/json' \
  -d '{"inputs":{"text":"Waterproof running shoes. <|image|> Grip test: <|video|>","image":"https://example.com/shoe.jpg","video":"/data/grip-test.mp4"},"dimensions":512}'

Use image, audio, or video dictionaries without a prompt for media-only inputs. On Google Cloud, the same inputs can be supplied in instances. The native vector has 768 dimensions; the model supports shortening it to 512, 256, or 128. Outputs are L2-normalized after truncation by default. /predict also accepts normalize: false to disable that final normalization.

🔵 Run sentence-transformers/all-MiniLM-L6-v2 on Azure AI

uv run hf-serve --model-id sentence-transformers/all-MiniLM-L6-v2 --task sentence-similarity --dtype float32 --cloud azure

🟠 Run sentence-transformers/all-MiniLM-L6-v2 on Amazon SageMaker

uv run hf-serve --model-id sentence-transformers/all-MiniLM-L6-v2 --task sentence-similarity --dtype float32 --cloud sagemaker

👂 Run facebook/wav2vec2-base-960h an automatic-speech-recognition model

uv run hf-serve --model-id facebook/wav2vec2-large-960h --task automatic-speech-recognition --dtype float16

And, then you can send a sample request as:

curl -L http://localhost:8080/predict \
    -H "Content-Type: application/json" \
    -d '{"inputs":"https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/1.flac"}'

Run voice activity detection

The voice-activity-detection task supports diarization-capable audio frame classification models whose processor exposes extract_speaker_dict. It returns speaker segments with start, end (seconds), and speaker (arrival-order index):

uv run hf-serve --model-id your-org/your-model --task voice-activity-detection

Send an audio URL, base64-encoded audio, or an audio file through the same JSON, form, or binary endpoints used by ASR:

curl -L http://localhost:8080/predict-json \
    -H "Content-Type: application/json" \
    -d '{"inputs":"https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/diarization_example.mp3"}'

Offline inference is the default. To process the recording in streaming chunks within one request, add "parameters":{"streaming_mode":"low_latency"} to the JSON body, or send streaming_mode=low_latency with a form upload. The other supported modes are very_low_latency and ultra_low_latency. Each request has its own speaker cache; the API does not retain a streaming session across requests.

On Google Cloud, run with --cloud google and send audio strings in instances. The same parameters apply to every instance, and the response contains one predictions entry per instance:

{"instances": ["<base64-encoded audio>"], "parameters": {"streaming_mode": "low_latency"}}

🔈 Run facebook/wav2vec2-lv-60-espeak-cv-ft (with phonemizer and espeak)

To run facebook/wav2vec2-lv-60-espeak-cv-ft on e.g. MacOS, you need to run the following:

DYLD_FALLBACK_LIBRARY_PATH=/opt/homebrew/lib uv run hf-serve --model-id facebook/wav2vec2-lv-60-espeak-cv-ft --task automatic-speech-recognition --dtype float16 --device mps

Note that if you have installed another version of ffmpeg with brew as e.g. brew install ffmpeg@7, you should use the following command instead:

DYLD_FALLBACK_LIBRARY_PATH=/opt/homebrew/opt/ffmpeg@7/lib uv run hf-serve --model-id facebook/wav2vec2-lv-60-espeak-cv-ft --task automatic-speech-recognition --dtype float16 --device mps

The main difference relies on the path used for DYLD_FALLBACK_LIBRARY_PATH which is now pointing to the exact brew-installed version of ffmpeg instead. More information on the compatibility issues with ffmpeg, torchcodec and torch at https://github.com/meta-pytorch/torchcodec?tab=readme-ov-file#installing-torchcodec.