Experimental version of inference

🚀 Introducing inference-exp - the evolution of inference

At Roboflow, we’re taking a bold step toward a new generation of inference — designed to be faster, more reliable, and more user-friendly. With this vision in mind, we’re building a new library called inference-exp.

This is an early-stage project, and we’re sharing initial versions to gather valuable community feedback. Your input will help us shape and steer this initiative in the right direction.

We’re excited to have you join us on this journey — let’s build something great together! 🤝

[!CAUTION] The inference-exp package is an experimental preview of upcoming inference capabilities.

• Features may change, break, or be removed without notice.
• We do not guarantee backward compatibility between releases.

❗ We strongly advise against using inference-exp in production systems - for such purposes please continue to use the stable inference package.

⚡ Installation

[!TIP] We recommend using uv to install inference-exp. To install the tool, follow official guide or use the snippet below:

curl -LsSf https://astral.sh/uv/install.sh | sh

Use the following command to install inference-exp on CPU machine 💻 (below you can find more advanced options):

uv pip install inference-exp
# or - if you use pip
pip install inference-exp

uv pip install "inference-exp[torch-cu128,onnx-cu12,trt10]" "tensorrt==10.12.0.36"
# or - if you use pip
pip install "inference-exp[torch-cu128,onnx-cu12,trt10]" "tensorrt==10.12.0.36"

# to install with ONNX backend
uv pip install "inference-exp[onnx-cpu]"
# or - to install only base dependencies
uv pip install inference-exp

📖 Basic Usage

from inference_exp import AutoModel
import cv2
import supervision as sv

# loads model from Roboflow API (loading from local dir also available)
model = AutoModel.from_pretrained("rfdetr-base")  
image = cv2.imread("<path-to-your-image>")
predictions = model(image)

# integration with supervision
annotator = sv.BoxAnnotator()
annotated = annotator.annotate(image.copy(), predictions[0].to_supervision())

[!TIP] Model failed to load, and you see error prompting you to install additional dependencies?

Take a look at 📜 Principles and Assumptions to understand why this happens and navigate to extras section to find out which extra dependency you need to install. The common issue is lack of ONNX backend required to run models trained on Roboflow platform.

📜 Principles and Assumptions

• We define a model as weights trained on a dataset, which can be exported or compiled into multiple equivalent model packages, each optimized for specific environments (e.g., speed, flexibility).

• The new inference library is multi-backend, able to run model packages in different formats depending on the installed dependencies - with the scope of supported models dependent on the choice of package extras made during installation

• We aim to keep the extra dependencies minimal while covering as broad a range of models as possible.

• By default, we include PyTorch and Hugging Face Transformers; optional extras are available for TensorRT (TRT) and ONNX backends, with a runtime preference order: TRT → Torch → ONNX. We wish new models are mostly based on Torch.

• Backend selection happens dynamically at runtime, based on model metadata and environment checks, but can be fully overridden by the user when needed.

🔌 Extra Dependencies

Extras dependencies are optional features of the package that can be installed with:

uv pip install "inference-exp[extras-name-1,extras-name-1]"
# or - if you use pip
pip install "inference-exp[extras-name-1,extras-name-2]"

In case of inference-exp, extras bring either additional backends (dependencies to run AI models of different type, like TensorRT engines) or additional models.

Backends

Additional models / capabilities

[!IMPORTANT]
Not all extras are possible to be installed together in a single environment. We try to make the extras as composable as possible, but this will not always be possible, and sometimes you need to choose which extras are to be installed.

🧠 Models

[!IMPORTANT] If you see a bug in model implementation or loading mechanism - create new issue tagging it with inference-exp-bug.

Additionally, We are working hard to extend pool of supported models - suggestions on new models to be added appreciated 🤝

Below there is a table showcasing models that are supported, with the hints regarding extra dependencies that are required.

Registered pre-trained weights

Below you can find a list of model IDs registered in Roboflow weights provider (along with notes about access rights).

• public-open - available without Roboflow API key, but under licenses for specific model

• public-api-key-gated - available for everyone with Roboflow API key

Models:

📜 Citations

@article{bolya2025PerceptionEncoder,
  title={Perception Encoder: The best visual embeddings are not at the output of the network},
  author={Daniel Bolya and Po-Yao Huang and Peize Sun and Jang Hyun Cho and Andrea Madotto and Chen Wei and Tengyu Ma and Jiale Zhi and Jathushan Rajasegaran and Hanoona Rasheed and Junke Wang and Marco Monteiro and Hu Xu and Shiyu Dong and Nikhila Ravi and Daniel Li and Piotr Doll{\'a}r and Christoph Feichtenhofer},
  journal={arXiv:2504.13181},
  year={2025}
}