tei-serving 1.1.0

Config-driven launcher for Hugging Face Text Embeddings Inference services

tei-serving

tei-serving is a config-driven launcher for Hugging Face Text Embeddings Inference (TEI). It wraps the TEI router with a small Python entrypoint that reads YAML, validates settings, applies local embedder compatibility patches when needed, and starts TEI with the generated CLI arguments.

What it does

Current capabilities:

1. Load an embedder or reranker YAML config.
2. Convert typed settings into TEI command-line arguments.
3. Patch local SentenceTransformers embedder exports into a TEI-compatible temporary copy.
4. Start the TEI router process.
5. Leave reranker configs untouched by embedder-specific patching.

Core files:

• src/tei_serving/__init__.py: runner and metadata patching.
• src/tei_serving/settings.py: config models and CLI serialization.
• src/tei_serving/main.py: CLI entrypoint.

Requirements

• Python >=3.12,<3.13 for local development.
• Docker for the production image.
• A TEI base image containing text-embeddings-router.
• GPU runtime when serving CUDA TEI images.

Installation

Production dependencies:

make install

Development environment:

make dev-install

Equivalent uv command:

uv sync --group dev --all-extras

Configuration

Embedder example:

kind: embedder

model:
  model-id: /models/testb
  dtype: float32
  pooling: cls

server:
  hostname: 0.0.0.0
  port: 8082

batching:
  max-batch-tokens: 16384
  max-client-batch-size: 32

Reranker example:

kind: reranker

model:
  model-id: BAAI/bge-reranker-base
  dtype: float16

server:
  hostname: 0.0.0.0
  port: 8081

batching:
  max-batch-tokens: 16384
  max-client-batch-size: 32

Running

Local CLI, assuming TEI is available on PATH:

tei-serving --config configs/embedder.yaml

Equivalent:

python -m tei_serving.main --config configs/embedder.yaml

Docker build:

docker build -t tei-serving:local .

Docker run with a mounted config:

docker run --gpus all --rm \
  -v /absolute/path/to/config.yaml:/config/config.yaml:ro \
  -v /absolute/path/to/models:/models:ro \
  tei-serving:local \
  --config /config/config.yaml

Local Embedder Patching

When kind: embedder, the runner prepares a temporary TEI-compatible copy before startup. Local model directories are copied directly. Hugging Face model IDs such as BAAI/bge-base-en-v1.5 are first downloaded with huggingface_hub.snapshot_download, then copied into the same private tei-serving-patched-models-* directory. Reranker configs skip this embedder-only patching.

The patch step updates metadata files TEI parses strictly:

• modules.json: normalizes SentenceTransformers module class paths.
• sentence_bert_config.json: adds max_seq_length when it can be inferred from config.json.
• 1_Pooling/config.json: adds explicit legacy pooling booleans.

The original model directory is never modified.

Development

make format
make lint
make type-check
make security
make test
make test-cov
make ci

Repository Layout

src/tei_serving/
  __init__.py      # runner and local embedder patching
  exceptions.py    # package exceptions
  main.py          # CLI entrypoint
  settings.py      # Pydantic settings and CLI serialization
configs/
  embedder.yaml    # example embedder config
  reranker.yaml    # example reranker config
tests/
  unit/            # unit tests for settings and runner behavior