Skip to main content

tei-serving

tei-serving is a config-driven launcher for Hugging Face Text Embeddings Inference (TEI). It wraps the TEI router with a small Python entrypoint that reads YAML, validates settings, and starts TEI with the generated CLI arguments.

What it does

Current capabilities:

  1. Load an embedder or reranker YAML config.
  2. Convert typed settings into TEI command-line arguments.
  3. Start the TEI router process with the configured model unchanged.

Core files:

Requirements

  • Python >=3.12,<3.13 for local development.
  • Docker for the production image.
  • A TEI base image containing text-embeddings-router.
  • GPU runtime when serving CUDA TEI images.

Installation

Production dependencies:

make install

Development environment:

make dev-install

Equivalent uv command:

uv sync --group dev --all-extras

Configuration

Embedder example:

kind: embedder

model:
  model-id: /models/testb
  dtype: float32
  pooling: cls

server:
  hostname: 0.0.0.0
  port: 8082

batching:
  max-batch-tokens: 16384
  max-client-batch-size: 32

Reranker example:

kind: reranker

model:
  model-id: BAAI/bge-reranker-base
  dtype: float16

server:
  hostname: 0.0.0.0
  port: 8081

batching:
  max-batch-tokens: 16384
  max-client-batch-size: 32

Running

Local CLI, assuming TEI is available on PATH:

tei-serving --config configs/embedder.yaml

Equivalent:

python -m tei_serving.main --config configs/embedder.yaml

Docker build:

docker build -t tei-serving:local .

Docker run with a mounted config:

docker run --gpus all --rm \
  -v /absolute/path/to/config.yaml:/config/config.yaml:ro \
  -v /absolute/path/to/models:/models:ro \
  tei-serving:local \
  --config /config/config.yaml

Model Serving

The runner passes the configured model-id directly to TEI for both embedder and reranker configs. It does not copy local model directories, download Hugging Face snapshots, or rewrite SentenceTransformers metadata before startup.

Development

make format
make lint
make type-check
make security
make test
make test-cov
make ci

Repository Layout

src/tei_serving/
  __init__.py      # runner implementation
  exceptions.py    # package exceptions
  main.py          # CLI entrypoint
  settings.py      # Pydantic settings and CLI serialization
configs/
  embedder.yaml    # example embedder config
  reranker.yaml    # example reranker config
tests/
  unit/            # unit tests for settings and runner behavior

Metadata

Release files for tei-serving 1.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for tei-serving 1.2.0
File Size Uploaded Provenance Actions
tei_serving-1.2.0.tar.gz 4.1 kB Without provenance Details for tei_serving-1.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for tei-serving 1.2.0
File Interpreter ABI Platform Actions
tei_serving-1.2.0-py3-none-any.whl Python 3 none any Details for tei_serving-1.2.0-py3-none-any.whl

Total release size: 9.6 kB

Release files / tei_serving-1.2.0.tar.gz

Download URL tei_serving-1.2.0.tar.gz
Size 4.1 kB
Tags Source
SHA-256 checksum
How to use checksums
c7e794ef159884845cca49a362b358d997ccb3dbd7f596590cd66d1c939f8eb7
BLAKE2b-256 checksum
How to use checksums
963327a816a976154eea189cc4b6acaaaf42458704b13ad444f854f00b175d5f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.11.19 {"installer":{"name":"uv","version":"0.11.19","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"12","id":"bookworm","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / tei_serving-1.2.0-py3-none-any.whl

Download URL tei_serving-1.2.0-py3-none-any.whl
Size 5.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
6116ba026f7feb85e515cbcd477e4c8a9e38e63da27225dd36407e773ac4a2d0
BLAKE2b-256 checksum
How to use checksums
65cc5f136141aaeaec4ec0470fe82b0d80485835f5e2c93ea29905c9b0197414
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.11.19 {"installer":{"name":"uv","version":"0.11.19","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"12","id":"bookworm","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for tei-serving 1.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page