Infinity is a high-throughput, low-latency REST API for serving vector embeddings, supporting a wide range of sentence-transformer models and frameworks.
Project description
Infinity ♾️
Infinity is a high-throughput, low-latency REST API for serving vector embeddings, supporting a wide range of sentence-transformer models and frameworks. Infinity is developed under MIT Licence: https://github.com/michaelfeil/infinity
Why Infinity:
Infinity provides the following features:
- Deploy virtually any SentenceTransformer - deploy the model you know from SentenceTransformers
- Fast inference backends: The inference server is built on top of torch, fastembed(onnx-cpu) and CTranslate2, getting most out of your CUDA or CPU hardware.
- Dynamic batching: New embedding requests are queued while GPU is busy with the previous ones. New requests are squeezed intro your GPU/CPU as soon as ready.
- Correct and tested implementation: Unit and end-to-end tested. Embeddings via infinity are identical to SentenceTransformers (up to numerical precision). Lets API users create embeddings till infinity and beyond.
- Easy to use: The API is built on top of FastAPI, Swagger makes it fully documented. API are aligned to OpenAI's Embedding specs. See below on how to get started.
Infinity demo:
In this gif below, we use sentence-transformers/all-MiniLM-L6-v2, deployed at batch-size=2. After initialization, from a second terminal 3 requests (payload 1,1,and 5 sentences) are sent via cURL.
Getting started
Install via pip
pip install infinity-emb[all]
Install from source with Poetry
Advanced: To install via Poetry use Poetry 1.6.1, Python 3.10 on Ubuntu 22.04
git clone https://github.com/michaelfeil/infinity
cd infinity
cd libs/infinity_emb
poetry install --extras all
Launch via Python
from infinity_emb import create server
create_server()
or launch the create_server()
command via CLI
infinity_emb --help
or launch the CLI using a pre-built docker container
model=BAAI/bge-small-en-v1.5
port=8080
docker run -it --gpus all -p $port:$port michaelf34/infinity:latest --model-name-or-path $model --port $port
The download path at runtime, can be controlled via the environment variable SENTENCE_TRANSFORMERS_HOME
.
Launch FAQ:
What are embedding models?
Embedding models can map any text to a low-dimensional dense vector which can be used for tasks like retrieval, classification, clustering, or semantic search. And it also can be used in vector databases for LLMs.The most know architecture are encoder-only transformers such as BERT, and most popular implementation include SentenceTransformers.
What models are supported?
All models of the sentence transformers org are supported https://huggingface.co/sentence-transformers / sbert.net. LLM's like LLAMA2-7B are not intended for deployment.
With the command --engine torch
the model must be compatible with https://github.com/UKPLab/sentence-transformers/.
- only models from Huggingface are supported.
With the command --engine ctranslate2
- only BERT
models are supported.
- only models from Huggingface are supported.
For the latest trends, you might want to check out one of the folloing models. https://huggingface.co/spaces/mteb/leaderboard
Launching multiple models in one dockerfile
Multiple models on one GPU is in experimental mode. You can use the following temporary solution:
FROM michaelf34/infinity:latest
# Dockerfile-ENTRYPOINT for multiple models via multiple ports
ENTRYPOINT ["/bin/sh", "-c", \
"(. /app/.venv/bin/activate && infinity_emb --port 8080 --model-name-or-path sentence-transformers/all-MiniLM-L6-v2 &);\
(. /app/.venv/bin/activate && infinity_emb --port 8081 --model-name-or-path intfloat/e5-large-v2 )"]
You can build and run it via:
docker build -t custominfinity . && docker run -it --gpus all -p 8080:8080 -p 8081:8081 custominfinity
Both models now run on two instances in one dockerfile servers.
Documentation
After startup, the Swagger Ui will be available under {url}:{port}/docs
, in this case http://localhost:8080/docs
.
Contribute and Develop
Install via Poetry 1.6.1 and Python3.10 on Ubuntu 22.04
cd libs/infinity_emb
poetry install --extras all --with test
To pass the CI:
cd libs/infinity_emb
make format
make lint
poetry run pytest ./tests
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for infinity_emb-0.0.5-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 42b7ab75320f78986b9d52e148a9c7f187b33aae5d83bb87fe861bed9b0e9231 |
|
MD5 | 8a3f810ac2834191aedcc916b4a1f194 |
|
BLAKE2b-256 | 5c2771977d6029beb9a3206764be4c6506fb538bd30ee052cb31630521e85f52 |