HuggingFace runtime for MLServer

This package provides a MLServer runtime compatible with HuggingFace Transformers.

Usage

You can install the runtime, alongside mlserver, as:

pip install mlserver mlserver-huggingface

For further information on how to use MLServer with HuggingFace, you can check out this worked out example.

Content Types

The HuggingFace runtime will always decode the input request using its own built-in codec. Therefore, content type annotations at the request level will be ignored. Note that this doesn't include input-level content type annotations, which will be respected as usual.

Settings

The HuggingFace runtime exposes a couple extra parameters which can be used to customise how the runtime behaves. These settings can be added under the parameters.extra section of your model-settings.json file, e.g.

---
emphasize-lines: 5-8
---
{
  "name": "qa",
  "implementation": "mlserver_huggingface.HuggingFaceRuntime",
  "parameters": {
    "extra": {
      "task": "question-answering",
      "optimum_model": true
    }
  }
}

These settings can also be injected through environment variables prefixed with `MLSERVER_MODEL_HUGGINGFACE_`, e.g.

```bash
MLSERVER_MODEL_HUGGINGFACE_TASK="question-answering"
MLSERVER_MODEL_HUGGINGFACE_OPTIMUM_MODEL=true
```

Loading models

Local models

It is possible to load a local model into a HuggingFace pipeline by specifying the model artefact folder path in parameters.uri in model-settings.json.

HuggingFace models

Models in the HuggingFace hub can be loaded by specifying their name in parameters.extra.pretrained_model in model-settings.json.

If `parameters.extra.pretrained_model` is specified, it takes precedence over `parameters.uri`.

Reference

You can find the full reference of the accepted extra settings for the HuggingFace runtime below:

.. autopydantic_settings:: mlserver_huggingface.settings.HuggingFaceSettings

Metadata

Release files for mlserver-huggingface 1.7.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for mlserver-huggingface 1.7.1
File	Size	Uploaded
mlserver_huggingface-1.7.1.tar.gz	15.4 kB	Jun 6, 2025	Details

Built distribution (wheel)

Table of built distributions (wheels) for mlserver-huggingface 1.7.1
File	Interpreter	ABI	Platform	Reset
mlserver_huggingface-1.7.1-py3-none-any.whl 21.4 kB Jun 6, 2025	Python 3	none	any	Details

Total release size: 36.8 kB

Release files / mlserver_huggingface-1.7.1.tar.gz

Download URL	mlserver_huggingface-1.7.1.tar.gz
Size	15.4 kB
Tags	Source
SHA-256 checksum How to use checksums	`3299b8526dfbbfbdb350dca34545328d0dac3cc8710549a8f60c332845cec9fa`
BLAKE2b-256 checksum How to use checksums	`3dcecf39b6e124ca20e51fced16ab9b2d42b1b0d9accef3f0a917aff6d8e10d4`
Upload date	Jun 6, 2025
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`poetry/2.1.3 CPython/3.10.17 Linux/6.11.0-1015-azure`

Release files / mlserver_huggingface-1.7.1-py3-none-any.whl

Download URL	mlserver_huggingface-1.7.1-py3-none-any.whl
Size	21.4 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`027e36fef0b11853cdd0982ce5fa9322773f2f0e49537a73c2db6c4013af767f`
BLAKE2b-256 checksum How to use checksums	`20fdf0be7ad050c160bbdf433c045dc89a96e9029a843cbf8dcb754f30d6c387`
Upload date	Jun 6, 2025
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`poetry/2.1.3 CPython/3.10.17 Linux/6.11.0-1015-azure`