Skip to main content

Contributors Forks Stargazers Issues License

Infinity Python / v2 ♾️

Requires Python 3.11–3.13. CPU and NVIDIA containers use Ubuntu 24.04 and Python 3.12.

Looking for the native Rust/LibTorch server? See Infinity v3 and its quickstart. This page documents the Python / v2 server.

codecov ci Downloads DOI Docker pulls

Infinity is a high-throughput, low-latency REST API for serving text-embeddings, reranking models, clip, clap and colpali. The Python packages use the MIT License. Infinity V3 uses the Infinity Restricted MIT License 1.0, with retained third-party and earlier grants.

Infinity V3 supports packed embedding tensors with shape and dtype metadata, including token matrices for late interaction.

Why Infinity

  • Deploy any model from HuggingFace: deploy any embedding, reranking, clip and sentence-transformer model from HuggingFace
  • Fast inference backends: The inference server is built on top of PyTorch, optimum (ONNX/TensorRT) and CTranslate2, using FlashAttention to get the most out of your NVIDIA CUDA, AMD ROCM, CPU, AWS INF2 or APPLE MPS accelerator. Infinity uses dynamic batching and tokenization dedicated in worker threads.
  • Multi-modal and multi-model: Mix-and-match multiple models. Infinity orchestrates them.
  • Tested implementation: Unit and end-to-end tested. Embeddings via infinity are correctly embedded. Lets API users create embeddings till infinity and beyond.
  • Easy to use: Built on FastAPI. Infinity CLI v2 allows launching of all arguments via Environment variable or argument. OpenAPI aligned to OpenAI's API specs. View the docs at https://michaelfeil.github.io/infinity on how to get started.

Logo Baseten.co Logo Runpod Logo TrueFoundry Logo Vast Logo DataGuard Logo SAP Logo Nosible Logo FreshWorks Logo Dstack Logo JamAI Logo Alibaba Group Logo BentoML Logo JinaAi Logo Dwarves Foundation Logo HF Logo Gradient.ai

Latest News 🔥

  • [2025/07] Blackwell support
  • [2024/11] AMD, CPU, ONNX docker images
  • [2024/10] pip install infinity_client
  • [2024/07] Inference deployment example via Modal and a free GPU deployment
  • [2024/06] Support for multi-modal: clip, text-classification & launch all arguments from env variables
  • [2024/05] launch multiple models using the v2 cli, including --api-key
  • [2024/03] infinity supports experimental int8 (cpu/cuda) and fp8 (H100/MI300) support
  • [2024/03] Docs are online: https://michaelfeil.github.io/infinity/latest/
  • [2024/02] Community meetup at the Run:AI Infra Club
  • [2024/01] TensorRT / ONNX inference
  • [2023/10] Initial release

Getting started

Launch the cli via pip install

pip install infinity-emb[all]

After your pip install, with your venv active, you can run the CLI directly.

infinity_emb v2 --model-id BAAI/bge-small-en-v1.5

Check the v2 --help command to get a description for all parameters.

infinity_emb v2 --help

Instead of installing the CLI via pip, you may also use docker to run michaelf34/infinity. Make sure you mount your accelerator ( i.e. install nvidia-docker and activate with --gpus all).

port=7997
model1=michaelfeil/bge-small-en-v1.5
model2=mixedbread-ai/mxbai-rerank-xsmall-v1
volume=$PWD/data

docker run -it --gpus all \
 -v $volume:/app/.cache \
 -p $port:$port \
 michaelf34/infinity:latest \
 v2 \
 --model-id $model1 \
 --model-id $model2 \
 --port $port

The cache path inside the docker container is set by the environment variable HF_HOME.

Specialized docker images

Docker container for CPU Use the `latest-cpu` image or `x.x.x-cpu` for slimer image. Run like any other cpu-only docker image. Optimum/Onnx is often the preferred engine.
docker run -it \
-v $volume:/app/.cache \
-p $port:$port \
michaelf34/infinity:latest-cpu \
v2 \
--engine optimum \
--model-id $model1 \
--model-id $model2 \
--port $port
Docker Container for ROCm (MI200 Series and MI300 Series) Use the `latest-rocm` image or `x.x.x-rocm` for rocm compatible inference. **This image is currently not build via CI/CD (to large), consider pinning to exact version.** Make sure you have ROCm is correctly installed and ready to use with Docker.

Visit Docs for more info.

Docker Container for Onnx-GPU, Cuda Extensions, TensorRT Use the `latest-trt-onnx` image or `x.x.x-trt-onnx` for nvidia compatible inference. **This image is currently not build via CI/CD (to large), consider pinning to exact version.**

This image has support for:

  • ONNX-Cuda "CudaExecutionProvider"
  • ONNX-TensorRT "TensorRTExecutionProvider" (may not always work due to version mismatch with ORT)
  • CudaExtensions and packages, e.g. Tri-Dao's pip install flash-attn package when using Pytorch.
  • nvcc compiler support
docker run -it \
-v $volume:/app/.cache \
-p $port:$port \
michaelf34/infinity:latest-trt-onnx \
v2 \
--engine optimum \
--device cuda \
--model-id $model1 \
--port $port

Using local models with Docker container

In order to deploy a local model with a docker container, you need to mount the model inside the container and specify the path in the container to the launch command.

Example:

git lfs install
cd /tmp
mkdir models && cd models && git clone https://huggingface.co/BAAI/bge-small-en-v1.5
docker run -it   -v /tmp/models:/models  -p 8081:8081  michaelf34/infinity:latest v2  --model-id "/models/bge-small-en-v1.5" --port 8081

Advanced CLI usage

Launching multiple models at once

Since infinity_emb>=0.0.34, you can use cli v2 method to launch multiple models at the same time. Checkout infinity_emb v2 --help for all args and validation.

Multiple Model CLI Playbook:

    1. cli options can be repeated e.g. v2 --model-id model/id1 --model-id model/id2 --batch-size 8 --batch-size 4. This will create two models model/id1 and model/id2
    1. or adapt the defaults by setting ENV Variables separated by ;: INFINITY_MODEL_ID="model/id1;model/id2;" && INFINITY_BATCH_SIZE="8;4;"
    1. single items are broadcasted to --model-id length, v2 --model-id model/id1 --model-id/id2 --batch-size 8 making both models have batch-size 8.
    1. Everything is broadcasted to the number of --model-id + API requests are routed to the --served-model-name/--model-id
Using environment variables instead of the cli All CLI arguments are also launchable via environment variables.

Environment variables start with INFINITY_{UPPER_CASE_SNAKE_CASE} and often match the --{lower-case-kebab-case} cli arguments.

The following two are equivalent:

  • CLI infinity_emb v2 --model-id BAAI/bge-base-en-v1.5
  • ENV-CLI: export INFINITY_MODEL_ID="BAAI/bge-base-en-v1.5" && infinity_emb v2

Multiple arguments can be used via ; syntax: INFINITY_MODEL_ID="model/id1;model/id2;"

API Key Supply an `--api-key secret123` via CLI or ENV INFINITY_API_KEY="secret123".
Choosing the fastest engine

With the command --engine torch the model must be compatible with https://github.com/UKPLab/sentence-transformers/ and AutoModel

With the command --engine optimum, there must be an onnx file. Models from https://huggingface.co/Xenova are recommended.

With the command --engine ctranslate2 - only BERT models are supported.

Telemetry opt-out

See which telemetry is collected: https://michaelfeil.eu/infinity/main/telemetry/

# Disable
export INFINITY_ANONYMOUS_USAGE_STATS="0"

Supported Tasks and Models by Infinity

Infinity aims to be the inference server supporting most functionality for embeddings, reranking and related RAG tasks. The following Infinity tests 15+ architectures and all of the below cases in the Github CI. Click on the sections below to find tasks and validated example models.

Text Embeddings

Text embeddings measure the relatedness of text strings. Embeddings are used for search, clustering, recommendations. Think about a private deployed version of openai's text embeddings. https://platform.openai.com/docs/guides/embeddings

Tested embedding models:

Other models:

Reranking Given a query and a list of documents, Reranking indexes the documents from most to least semantically relevant to the query. Think like a locally deployed version of https://docs.cohere.com/reference/rerank

Tested reranking models:

Other reranking models:

Multi-modal and cross-modal - image and audio embeddings Specialized embedding models that allow for image<->text or image<->audio search. Typically, these models allow for text<->text, text<->other and other<->other search, with accuracy tradeoffs when going cross-modal.

Image<->text models can be used for e.g. photo-gallery search, where users can type in keywords to find photos, or use a photo to find related images. Audio<->text models are less popular, and can be e.g. used to find music songs based on a text description or related music songs.

Tested image<->text models:

Tested audio<->text models:

  • Clap Models from LAION
  • limited number open source organizations training these models
    • Note: The sampling rate of the audio data needs to match the model *

Not supported:

  • Plain vision models e.g. nomic-ai/nomic-embed-vision-v1.5
ColBert-style late-interaction Embeddings ColBert Embeddings don't perform any special Pooling methods, but return the raw **token embeddings**. The **token embeddings** are then to be scored with the MaxSim Metric in a VectorDB (Qdrant / Vespa)

For usage via the RestAPI, late-interaction embeddings may best be transported via base64 encoding. Example notebook: https://colab.research.google.com/drive/14FqLc0N_z92_VgL_zygWV5pJZkaskyk7?usp=sharing

Tested colbert models:

ColPali-style late-interaction Image<->Text Embeddings Similar usage to ColBert, but scanning over an image<->text instead of only text.

For usage via the RestAPI, late-interaction embeddings may best be transported via base64 encoding. Example notebook: https://colab.research.google.com/drive/14FqLc0N_z92_VgL_zygWV5pJZkaskyk7?usp=sharing

Tested ColPali/ColQwen models:

Text classification A bert-style multi-label text classification. Classifies it into distinct categories.

Tested models:

Infinity usage via the Python API

Instead of the cli & RestAPI use infinity's interface via the Python API. This gives you most flexibility. The Python API builds on asyncio with its await/async features, to allow concurrent processing of requests. Arguments of the CLI are also available via Python.

Embeddings

import asyncio
from infinity_emb import AsyncEngineArray, EngineArgs, AsyncEmbeddingEngine

sentences = ["Embed this is sentence via Infinity.", "Paris is in France."]
array = AsyncEngineArray.from_args([
  EngineArgs(model_name_or_path = "BAAI/bge-small-en-v1.5", engine="torch", embedding_dtype="float32", dtype="auto")
])

async def embed_text(engine: AsyncEmbeddingEngine):
    async with engine:
        embeddings, usage = await engine.embed(sentences=sentences)
    # or handle the async start / stop yourself.
    await engine.astart()
    embeddings, usage = await engine.embed(sentences=sentences)
    await engine.astop()
asyncio.run(embed_text(array[0]))

Reranking

Reranking gives you a score for similarity between a query and multiple documents. Use it in conjunction with a VectorDB+Embeddings, or as standalone for small amount of documents. Please select a model from huggingface that is a AutoModelForSequenceClassification compatible model with one class classification.

import asyncio
from infinity_emb import AsyncEngineArray, EngineArgs, AsyncEmbeddingEngine
query = "What is the python package infinity_emb?"
docs = ["This is a document not related to the python package infinity_emb, hence...",
    "Paris is in France!",
    "infinity_emb is a package for sentence embeddings and rerankings using transformer models in Python!"]
array = AsyncEmbeddingEngine.from_args(
  [EngineArgs(model_name_or_path = "mixedbread-ai/mxbai-rerank-xsmall-v1", engine="torch")]
)

async def rerank(engine: AsyncEmbeddingEngine):
    async with engine:
        ranking, usage = await engine.rerank(query=query, docs=docs)
        print(list(zip(ranking, docs)))
    # or handle the async start / stop yourself.
    await engine.astart()
    ranking, usage = await engine.rerank(query=query, docs=docs)
    await engine.astop()

asyncio.run(rerank(array[0]))

When using the CLI, use this command to launch rerankers:

infinity_emb v2 --model-id mixedbread-ai/mxbai-rerank-xsmall-v1

Image-Embeddings: CLIP models

CLIP models are able to encode images and text at the same time.

import asyncio
from infinity_emb import AsyncEngineArray, EngineArgs, AsyncEmbeddingEngine

sentences = ["This is awesome.", "I am bored."]
images = ["http://images.cocodataset.org/val2017/000000039769.jpg"]
engine_args = EngineArgs(
    model_name_or_path = "wkcn/TinyCLIP-ViT-8M-16-Text-3M-YFCC15M",
    engine="torch"
)
array = AsyncEngineArray.from_args([engine_args])

async def embed(engine: AsyncEmbeddingEngine):
    await engine.astart()
    embeddings, usage = await engine.embed(sentences=sentences)
    embeddings_image, _ = await engine.image_embed(images=images)
    await engine.astop()

asyncio.run(embed(array["wkcn/TinyCLIP-ViT-8M-16-Text-3M-YFCC15M"]))

Audio-Embeddings: CLAP models

CLAP models are able to encode audio and text at the same time.

import asyncio
from infinity_emb import AsyncEngineArray, EngineArgs, AsyncEmbeddingEngine
import requests
import soundfile as sf
import io

sentences = ["This is awesome.", "I am bored."]

url = "https://bigsoundbank.com/UPLOAD/wav/2380.wav"
raw_bytes = requests.get(url, stream=True).content

audios = [raw_bytes]
engine_args = EngineArgs(
    model_name_or_path = "laion/clap-htsat-unfused",
    dtype="float32",
    engine="torch"

)
array = AsyncEngineArray.from_args([engine_args])

async def embed(engine: AsyncEmbeddingEngine):
    await engine.astart()
    embeddings, usage = await engine.embed(sentences=sentences)
    embedding_audios = await engine.audio_embed(audios=audios)
    await engine.astop()

asyncio.run(embed(array["laion/clap-htsat-unfused"]))

Text Classification

Use text classification with Infinity's classify feature, which allows for sentiment analysis, emotion detection, and more classification tasks.

import asyncio
from infinity_emb import AsyncEngineArray, EngineArgs, AsyncEmbeddingEngine

sentences = ["This is awesome.", "I am bored."]
engine_args = EngineArgs(
    model_name_or_path = "SamLowe/roberta-base-go_emotions",
    engine="torch", model_warmup=True)
array = AsyncEngineArray.from_args([engine_args])

async def classifier(engine: AsyncEmbeddingEngine):
    async with engine:
        predictions, usage = await engine.classify(sentences=sentences)
    # or handle the async start / stop yourself.
    await engine.astart()
    predictions, usage = await engine.classify(sentences=sentences)
    await engine.astop()
asyncio.run(classifier(array["SamLowe/roberta-base-go_emotions"]))

Infinity usage via the Python Client

Infinity has a generated client code for RestAPI client side usage.

If you want to call a remote infinity instance via RestAPI, install the following package locally:

pip install infinity_client

For more information, check out the Client Readme https://github.com/michaelfeil/infinity/tree/main/libs/client_infinity/infinity_client

Integrations:

Documentation

View the docs at https:///michaelfeil.github.io/infinity on how to get started. After startup, the Swagger Ui will be available under {url}:{port}/docs, in this case http://localhost:7997/docs. You can also find a interactive preview here: https://infinity.modal.michaelfeil.eu/docs (and https://michaelfeil-infinity.hf.space/docs)

Contribute and Develop

Install via Poetry 2.5.1, Python 3.12 on Ubuntu 24.04

cd libs/infinity_emb
poetry install --extras all --with lint,test

To pass the CI:

cd libs/infinity_emb
make precommit

Contributions to the Python packages must be compatible with the MIT License. Infinity V3 contributions follow its contribution policy and license.

Citation

@software{feil_2023_11630143,
  author       = {Feil, Michael},
  title        = {Infinity - To Embeddings and Beyond},
  month        = oct,
  year         = 2023,
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.11630143},
  url          = {https://doi.org/10.5281/zenodo.11630143}
}

💚 Current contributors

Metadata

Release files for infinity-emb 0.0.80

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for infinity-emb 0.0.80
File Size Uploaded Provenance Actions
infinity_emb-0.0.80.tar.gz 83.7 kB With provenance Details for infinity_emb-0.0.80.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for infinity-emb 0.0.80
File Interpreter ABI Platform Actions
infinity_emb-0.0.80-py3-none-any.whl Python 3 none any Details for infinity_emb-0.0.80-py3-none-any.whl

Total release size: 184.0 kB

Release files / infinity_emb-0.0.80.tar.gz

Download URL infinity_emb-0.0.80.tar.gz
Size 83.7 kB
Tags Source
SHA-256 checksum
How to use checksums
e897a2e1ad728f1e885d0043cdedc01524923279c736b85d27c894f955f90959
BLAKE2b-256 checksum
How to use checksums
f359bdf0da24b314160871552dd8c9060a9023001947cc3041c7f017f4e6e843
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.

Transparency log

Release files / infinity_emb-0.0.80-py3-none-any.whl

Download URL infinity_emb-0.0.80-py3-none-any.whl
Size 100.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
da974c66d9723f7208ccfc76c1f79aea8b4b2987f36e8fb29b6afd8235f92e77
BLAKE2b-256 checksum
How to use checksums
7fdd66ce3652550f0dfc5a4ab3e82a36655375f756a86794dbc3054666d4a3cb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.

Transparency log

Security for infinity-emb 0.0.80

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for infinity_emb-0.0.80.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.0.80 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page