xinfer 0.0.3

A unified interface to run inference on machine learning libraries.

Explore the docs »
Feature Request · Report Bug · Discussions · About me

Why x.infer?

If you'd like to run many models from different libraries without having to rewrite your inference code, xinfer is for you. It has a simple API and is easy to extend. Currently supports Transformers, Ultralytics, and TIMM.

Have a custom machine learning model? Create a class that implements the BaseModel interface and register it with xinfer. See Adding New Models for more details.

Key Features

• Unified Interface: Interact with different machine learning models through a single, consistent API.
• Modular Design: Integrate and swap out models without altering the core framework.
• Ease of Use: Simplifies model loading, input preprocessing, inference execution, and output postprocessing.
• Extensibility: Add support for new models and libraries with minimal code changes.

Quickstart

Here's a quick example demonstrating how to use x.infer with a Transformers model:

import xinfer

model = xinfer.create_model("vikhyatk/moondream2")

image = "https://raw.githubusercontent.com/vikhyat/moondream/main/assets/demo-1.jpg"
prompt = "Describe this image. "

model.infer(image, prompt)

>>> An animated character with long hair and a serious expression is eating a large burger at a table, with other characters in the background.

Supported Libraries

• Hugging Face Transformers: Natural language processing models for tasks like text classification, translation, and summarization.
• Ultralytics YOLO: State-of-the-art real-time object detection models.
• Custom Models: Support for your own machine learning models and architectures.

Prerequisites

Install PyTorch.

Installation

Install x.infer using pip:

pip install xinfer

With specific libraries:

pip install "xinfer[transformers]"
pip install "xinfer[ultralytics]"
pip install "xinfer[timm]"

Or locally:

pip install -e .

With specific libraries:

pip install -e ."[transformers]"
pip install -e ."[ultralytics]"
pip install -e ."[timm]"

See example.ipynb for more examples.

Usage

Supported Models

Transformers:

• BLIP2 Series

model = xinfer.create_model("Salesforce/blip2-opt-2.7b")

• Moondream2

model = xinfer.create_model("vikhyatk/moondream2")

[!NOTE] Wish to load an unlisted model? You can load any Vision2Seq model from Transformers by using the Vision2SeqModel class.

from xinfer.transformers import Vision2SeqModel

model = Vision2SeqModel("facebook/chameleon-7b")
model = xinfer.create_model(model)

TIMM:

• EVA02 Series

model = xinfer.create_model("eva02_small_patch14_336.mim_in22k_ft_in1k")

[!NOTE] Wish to load an unlisted model? You can load any model from TIMM by using the TIMMModel class.

from xinfer.timm import TimmModel

model = TimmModel("resnet18")
model = xinfer.create_model(model)

Ultralytics:

• YOLOv8 Series

model = xinfer.create_model("yolov8n")

• YOLOv10 Series

model = xinfer.create_model("yolov10x")

• YOLOv11 Series

model = xinfer.create_model("yolov11s")

[!NOTE] Wish to load an unlisted model? You can load any model from Ultralytics by using the UltralyticsModel class.

from xinfer.ultralytics import UltralyticsModel

model = UltralyticsModel("yolov5n6u")
model = xinfer.create_model(model)

List Models

import xinfer

xinfer.list_models()

Available Models
Implementation	Model ID	Input --> Output
timm	eva02_large_patch14_448.mim_m38m_ft_in22k_in1k	image --> class
timm	eva02_large_patch14_448.mim_m38m_ft_in1k	image --> class
timm	eva02_large_patch14_448.mim_in22k_ft_in22k_in1k	image --> class
timm	eva02_large_patch14_448.mim_in22k_ft_in1k	image --> class
timm	eva02_base_patch14_448.mim_in22k_ft_in22k_in1k	image --> class
timm	eva02_base_patch14_448.mim_in22k_ft_in1k	image --> class
timm	eva02_small_patch14_336.mim_in22k_ft_in1k	image --> class
timm	eva02_tiny_patch14_336.mim_in22k_ft_in1k	image --> class
transformers	Salesforce/blip2-opt-6.7b-coco	image-text --> text
transformers	Salesforce/blip2-flan-t5-xxl	image-text --> text
transformers	Salesforce/blip2-opt-6.7b	image-text --> text
transformers	Salesforce/blip2-opt-2.7b	image-text --> text
transformers	vikhyatk/moondream2	image-text --> text
ultralytics	yolov8x	image --> objects
ultralytics	yolov8m	image --> objects
ultralytics	yolov8l	image --> objects
ultralytics	yolov8s	image --> objects
ultralytics	yolov8n	image --> objects
ultralytics	yolov10x	image --> objects
ultralytics	yolov10m	image --> objects
...
...

Adding New Models

• Step 1: Create a new model class that implements the BaseModel interface.

• Step 2: Implement the required abstract methods load_model, infer, and infer_batch.

• Step 3: Decorate your class with the register_model decorator, specifying the model ID, implementation, and input/output.

For example:

@xinfer.register_model("my-model", "custom", ModelInputOutput.IMAGE_TEXT_TO_TEXT)
class MyModel(BaseModel):
    def load_model(self):
        # Load your model here
        pass

    def infer(self, image, prompt):
        # Run single inference 
        pass

    def infer_batch(self, images, prompts):
        # Run batch inference here
        pass