hf-serve

Installation

First you need to setup your environment with uv (or with your preferred Python environment manager).

uv venv --python 3.12
source .venv/bin/activate

Install it from the uv.lock file for CPU / MPS as follows:

uv sync --active --frozen --extra cpu

Alternatively, install it on NVIDIA CUDA 12.6 as follows:

uv sync --active --frozen --extra cuda --extra flash-attn --preview-features extra-build-dependencies

Or if you're on CUDA 12.8 then:

uv sync --active --frozen --extra cuda-128 --extra flash-attn --preview-features extra-build-dependencies

Or for CUDA 13.0, including NVIDIA GB10 / DGX Spark:

uv sync --active --frozen --extra cuda-130

$ uv run hf-serve --help

Container images

On each push to main, CI publishes images to ghcr.io/<owner>/hf-serve using the first seven characters of the commit SHA as the tag. Images are built on native runners for the architectures listed below:

For example, to pull the CUDA 13.0 image published by the upstream repository:

docker pull ghcr.io/huggingface/hf-serve:<sha>-cu130-arm64

Use a commit published after multi-platform support was added; older tags may only contain AMD64 images. Docker selects the architecture for the host.

To build the CUDA 13.0 variant locally from this checkout:

docker buildx build --platform linux/arm64 \
  --build-arg BASE_IMAGE=nvidia/cuda:13.0.2-devel-ubuntu24.04 \
  --build-arg TORCH_EXTRA=cuda-130 \
  --tag hf-serve:gb10 --load .

Examples

🤏 Run HuggingFaceTB/SmolLM3-3B with an OpenAI API

uv run hf-serve --model-id HuggingFaceTB/SmolLM3-3B --task text-generation --dtype float16

🔵 Run sentence-transformers/all-MiniLM-L6-v2 on Azure AI

uv run hf-serve --model-id sentence-transformers/all-MiniLM-L6-v2 --task sentence-similarity --dtype float32 --cloud azure

🟠 Run sentence-transformers/all-MiniLM-L6-v2 on Amazon SageMaker

uv run hf-serve --model-id sentence-transformers/all-MiniLM-L6-v2 --task sentence-similarity --dtype float32 --cloud sagemaker

👂 Run facebook/wav2vec2-base-960h an automatic-speech-recognition model

uv run hf-serve --model-id facebook/wav2vec2-large-960h --task automatic-speech-recognition --dtype float16

And, then you can send a sample request as:

curl -L http://localhost:8080/predict \
    -H "Content-Type: application/json" \
    -d '{"inputs":"https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/1.flac"}'

Run voice activity detection

The voice-activity-detection task supports diarization-capable audio frame classification models whose processor exposes extract_speaker_dict. It returns speaker segments with start, end (seconds), and speaker (arrival-order index):

uv run hf-serve --model-id your-org/your-model --task voice-activity-detection

Send an audio URL, base64-encoded audio, or an audio file through the same JSON, form, or binary endpoints used by ASR:

curl -L http://localhost:8080/predict-json \
    -H "Content-Type: application/json" \
    -d '{"inputs":"https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/diarization_example.mp3"}'

Offline inference is the default. To process the recording in streaming chunks within one request, add "parameters":{"streaming_mode":"low_latency"} to the JSON body, or send streaming_mode=low_latency with a form upload. The other supported modes are very_low_latency and ultra_low_latency. Each request has its own speaker cache; the API does not retain a streaming session across requests.

On Google Cloud, run with --cloud google and send audio strings in instances. The same parameters apply to every instance, and the response contains one predictions entry per instance:

{"instances": ["<base64-encoded audio>"], "parameters": {"streaming_mode": "low_latency"}}

🔈 Run facebook/wav2vec2-lv-60-espeak-cv-ft (with phonemizer and espeak)

To run facebook/wav2vec2-lv-60-espeak-cv-ft on e.g. MacOS, you need to run the following:

DYLD_FALLBACK_LIBRARY_PATH=/opt/homebrew/lib uv run hf-serve --model-id facebook/wav2vec2-lv-60-espeak-cv-ft --task automatic-speech-recognition --dtype float16 --device mps

Note that if you have installed another version of ffmpeg with brew as e.g. brew install ffmpeg@7, you should use the following command instead:

DYLD_FALLBACK_LIBRARY_PATH=/opt/homebrew/opt/ffmpeg@7/lib uv run hf-serve --model-id facebook/wav2vec2-lv-60-espeak-cv-ft --task automatic-speech-recognition --dtype float16 --device mps

The main difference relies on the path used for DYLD_FALLBACK_LIBRARY_PATH which is now pointing to the exact brew-installed version of ffmpeg instead. More information on the compatibility issues with ffmpeg, torchcodec and torch at https://github.com/meta-pytorch/torchcodec?tab=readme-ov-file#installing-torchcodec.