Skip to main content

A unified interface to run inference on machine learning libraries.

Project description

Python PyPI version Downloads License OS Support


🤔 Why x.infer?

So, a new computer vision model just dropped last night. It's called GPT-54o-mini-vision-pro-max-xxxl. It's a super cool model, open-source, open-weights, open-data, all the good stuff.

You're excited. You want to try it out.

But it's written in a new framework, TyPorch that you know nothing about. You don't want to spend a weekend learning TyPorch just to find out the model is not what you expected.

This is where x.infer comes in.

x.infer is a simple library that allows you to run inference with any computer vision model in just a few lines of code. All in Python.

Out of the box, x.infer supports the following frameworks:

Transformers TIMM Ultralytics vLLM Ollama

Combined, x.infer supports over 1000+ models from all the above frameworks.

Tasks supported:

Image Classification Object Detection Image to Text

Run any supported model using the following 4 lines of code:

import xinfer

model = xinfer.create_model("vikhyatk/moondream2")
model.infer(image, prompt)         # Run single inference
model.infer_batch(images, prompts) # Run batch inference
model.launch_gradio()              # Launch Gradio interface

Have a custom model? Create a class that implements the BaseModel interface and register it with x.infer. See 🔧 Adding New Models for more details.

🌟 Key Features

x.infer
  • Unified Interface: Interact with different computer vision frameworks through a single, consistent API.
  • Modular Design: Integrate and swap out models without altering the core framework.
  • Extensibility: Add support for new models and libraries with minimal code changes.

🚀 Quickstart

Here's a quick example demonstrating how to use x.infer with a Transformers model:

Open In Colab Open In Kaggle

import xinfer

model = xinfer.create_model("vikhyatk/moondream2")

image = "https://raw.githubusercontent.com/vikhyat/moondream/main/assets/demo-1.jpg"
prompt = "Describe this image. "

model.infer(image, prompt)

>>> An animated character with long hair and a serious expression is eating a large burger at a table, with other characters in the background.

Get a list of models:

xinfer.list_models()
       Available Models                                      
┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓
┃ Implementation ┃ Model ID                                              ┃ Input --> Output     ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩
│ timm           │ timm/eva02_large_patch14_448.mim_m38m_ft_in22k_in1k   │ image --> categories │
│ timm           │ timm/eva02_large_patch14_448.mim_m38m_ft_in1k         │ image --> categories │
│ timm           │ timm/eva02_large_patch14_448.mim_in22k_ft_in22k_in1k  │ image --> categories │
│ timm           │ timm/eva02_large_patch14_448.mim_in22k_ft_in1k        │ image --> categories │
│ timm           │ timm/eva02_base_patch14_448.mim_in22k_ft_in22k_in1k   │ image --> categories │
│ timm           │ timm/eva02_base_patch14_448.mim_in22k_ft_in1k         │ image --> categories │
│ timm           │ timm/eva02_small_patch14_336.mim_in22k_ft_in1k        │ image --> categories │
│ timm           │ timm/eva02_tiny_patch14_336.mim_in22k_ft_in1k         │ image --> categories │
│ transformers   │ Salesforce/blip2-opt-6.7b-coco                        │ image-text --> text  │
│ transformers   │ Salesforce/blip2-flan-t5-xxl                          │ image-text --> text  │
│ transformers   │ Salesforce/blip2-opt-6.7b                             │ image-text --> text  │
│ transformers   │ Salesforce/blip2-opt-2.7b                             │ image-text --> text  │
│ transformers   │ fancyfeast/llama-joycaption-alpha-two-hf-llava        │ image-text --> text  │
│ transformers   │ vikhyatk/moondream2                                   │ image-text --> text  │
│ transformers   │ sashakunitsyn/vlrm-blip2-opt-2.7b                     │ image-text --> text  │
│ ultralytics    │ ultralytics/yolov8x                                   │ image --> boxes      │
│ ultralytics    │ ultralytics/yolov8m                                   │ image --> boxes      │
│ ultralytics    │ ultralytics/yolov8l                                   │ image --> boxes      │
│ ultralytics    │ ultralytics/yolov8s                                   │ image --> boxes      │
│ ultralytics    │ ultralytics/yolov8n                                   │ image --> boxes      │
│ ...            │ ...                                                   │ ...                  │
│ ...            │ ...                                                   │ ...                  │
└────────────────┴───────────────────────────────────────────────────────┴──────────────────────┘

If you're running in a Juypter Notebook environment, you can specify interactive=True to list and search supported models interactively.

https://github.com/user-attachments/assets/d51cf707-2001-478c-881a-ae27f690d1bc

🖥️ Gradio Demo for Supported Models

For all supported models, you can launch a Gradio interface to interact with the model. This is useful for quickly testing the model and visualizing the results.

Once the model is created, you can launch the Gradio interface with the following line of code:

model.launch_gradio()

https://github.com/user-attachments/assets/25ce31f3-c9e2-4934-b341-000a6d1b7dc4

If you'd like to launch a Gradio interface with all models available in a dropdown, you can use the following line of code:

xinfer.launch_gradio_demo()

https://github.com/user-attachments/assets/bd46f55a-573f-45b9-910f-e22bee27fd3d

See Gradio Demo for more details.

📦 Installation

[!IMPORTANT] You must have PyTorch installed to use x.infer.

To install the barebones x.infer (without any optional dependencies), run:

pip install xinfer

x.infer can be used with multiple optional dependencies. You'll just need to install one or more of the following:

pip install "xinfer[transformers]"
pip install "xinfer[ultralytics]"
pip install "xinfer[timm]"
pip install "xinfer[vllm]"

To install all optional dependencies, run:

pip install "xinfer[all]"

To install from a local directory, run:

git clone https://github.com/dnth/x.infer.git
cd x.infer
pip install -e .

🛠️ Usage

Supported Models

Transformers
Model Usage
BLIP2 Series
xinfer.create_model("Salesforce/blip2-opt-2.7b")
Moondream2
xinfer.create_model("vikhyatk/moondream2")
VLRM-BLIP2
xinfer.create_model("sashakunitsyn/vlrm-blip2-opt-2.7b")
JoyCaption
xinfer.create_model("fancyfeast/llama-joycaption-alpha-two-hf-llava")
Llama-3.2 Vision Series
xinfer.create_model("meta-llama/Llama-3.2-11B-Vision-Instruct")
Florence-2 Series
xinfer.create_model("microsoft/Florence-2-base-ft")

You can also load any AutoModelForVision2Seq model from Transformers by using the Vision2SeqModel class.

from xinfer.transformers import Vision2SeqModel

model = Vision2SeqModel("facebook/chameleon-7b")
model = xinfer.create_model(model)
TIMM

All models from TIMM fine-tuned for ImageNet 1k are supported.

For example load a resnet18.a1_in1k model:

xinfer.create_model("timm/resnet18.a1_in1k")

You can also load any model (or a custom timm model) by using the TIMMModel class.

from xinfer.timm import TimmModel

model = TimmModel("resnet18")
model = xinfer.create_model(model)
Ultralytics
Model Usage
YOLOv8 Series
xinfer.create_model("ultralytics/yolov8n")
YOLOv10 Series
xinfer.create_model("ultralytics/yolov10x")
YOLOv11 Series
xinfer.create_model("ultralytics/yolov11s")

You can also load any model from Ultralytics by using the UltralyticsModel class.

from xinfer.ultralytics import UltralyticsModel

model = UltralyticsModel("yolov5n6u")
model = xinfer.create_model(model)
vLLM
Model Usage
Molmo-72B
xinfer.create_model("vllm/allenai/Molmo-72B-0924")
Molmo-7B-D
xinfer.create_model("vllm/allenai/Molmo-7B-D-0924")
Molmo-7B-O
xinfer.create_model("vllm/allenai/Molmo-7B-O-0924")
Ollama

To use Ollama models, you'll need to install the Ollama on your machine. See Ollama Installation Guide for more details.

Model Usage
LLaVA Phi3
xinfer.create_model("ollama/llava-phi3")

🔧 Adding New Models

  • Step 1: Create a new model class that implements the BaseModel interface.

  • Step 2: Implement the required abstract methods load_model, infer, and infer_batch.

  • Step 3: Decorate your class with the register_model decorator, specifying the model ID, implementation, and input/output.

For example:

@register_model("my-model", "custom", ModelInputOutput.IMAGE_TEXT_TO_TEXT)
class MyModel(BaseModel):
    def load_model(self):
        # Load your model here
        pass

    def infer(self, image, prompt):
        # Run single inference 
        pass

    def infer_batch(self, images, prompts):
        # Run batch inference here
        pass

See an example implementation of the Molmo model here.

🤝 Contributing

If you'd like to contribute, here are some ways you can help:

  1. Add support for new models: Implement new model classes following the steps in the Adding New Models section.

  2. Improve documentation: Help us enhance our documentation, including this README, inline code comments, and the official docs.

  3. Report bugs: If you find a bug, please open an issue with a clear description and steps to reproduce.

  4. Suggest enhancements: Have ideas for new features? Open a feature request.

  5. Submit pull requests: Feel free to fork the repository and submit pull requests for any improvements you've made.

Please also see the code of conduct here. Thank you for helping make x.infer better!

⚠️ Disclaimer

x.infer is not affiliated with any of the libraries it supports. It is a simple wrapper that allows you to run inference with any of the supported models.

Although x.infer is Apache 2.0 licensed, the models it supports may have their own licenses. Please check the individual model repositories for more details.


Back to Top

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

xinfer-0.1.3.tar.gz (38.1 MB view details)

Uploaded Source

Built Distribution

xinfer-0.1.3-py2.py3-none-any.whl (44.4 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file xinfer-0.1.3.tar.gz.

File metadata

  • Download URL: xinfer-0.1.3.tar.gz
  • Upload date:
  • Size: 38.1 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.12.7

File hashes

Hashes for xinfer-0.1.3.tar.gz
Algorithm Hash digest
SHA256 a7b3138e586ce60ce5320c9efea1f140e37627152cc41926e2cef1846eb4a751
MD5 f282b306a5db5a1ee0c6bd2f06cb7693
BLAKE2b-256 2c55b43bb4a5f203cdba6a5d6a190e35a554d8e70c7b838c71e4478529d84156

See more details on using hashes here.

File details

Details for the file xinfer-0.1.3-py2.py3-none-any.whl.

File metadata

  • Download URL: xinfer-0.1.3-py2.py3-none-any.whl
  • Upload date:
  • Size: 44.4 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.12.7

File hashes

Hashes for xinfer-0.1.3-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 58c39744c56e38d41c233d56d2f9817c92b6795c49c88ca457b7176b1a0fcfb9
MD5 effa87be6946545f254f7b8f8541e601
BLAKE2b-256 efda3f06f6b27fdbb53abf2f5469f527a97663838b1c6c8a4911523346a846d1

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page