Embedders LitServe

Installation

pip install embedders-litserve

Running the server from this repository

The server itself is not part of the PyPI distribution. To run it from this repository, configure .env and install its optional dependencies:

uv sync --extra server
uv run --extra server python app.py

Client

import asyncio

from embedders_litserve import EmbedderClient


async def main() -> None:
    async with EmbedderClient("siglip2", "http://localhost:8000") as client:
        await client.verify_model()
        response = await client.embed_texts(["a red fox"])
        print(response.embeddings)


asyncio.run(main())

Benchmark (repository only)

Run a local HTTP throughput benchmark against a running server:

uv run --extra server python bench.py http://localhost:8000
uv run --extra server python bench.py http://localhost:8000 --batch-size 64 --requests 100 --concurrency 8

The CLI sends bundled test_images as base64 to /predict. A full endpoint URL and a custom --images-dir are also supported. Images are reused when needed. The reported images/sec includes HTTP transfer and response decoding, excludes local image loading and the warmup requests (--warmup, default 2), and counts only runs where every response contains the expected number of embeddings. Use --help for all options.

Each successful run saves a dated UTC JSON file in benchmark_history/ (gitignored). It contains the server's GET /info response (model, devices, workers and server configuration), benchmark parameters, timestamps, elapsed time and images/sec. The server must expose /info; metadata is fetched before warmup, outside the measurement. Use --history-dir to change the output directory.

Publishing

Create a GitHub release with a tag such as v1.2.3. The publish workflow sets the package version to 1.2.3, builds distributions, and publishes them via PyPI Trusted Publishing. Configure the PyPI project as a trusted publisher for this repository, workflow publish.yml, and environment pypi before the first release.