tei-serving
tei-serving is a config-driven launcher for Hugging Face Text Embeddings Inference (TEI). It wraps the TEI router with a small Python entrypoint that reads YAML, validates settings, and starts TEI with the generated CLI arguments.
What it does
Current capabilities:
- Load an embedder or reranker YAML config.
- Convert typed settings into TEI command-line arguments.
- Start the TEI router process with the configured model unchanged.
Core files:
src/tei_serving/__init__.py: runner implementation.src/tei_serving/settings.py: config models and CLI serialization.src/tei_serving/main.py: CLI entrypoint.
Requirements
- Python
>=3.12,<3.13for local development. - Docker for the production image.
- A TEI base image containing
text-embeddings-router. - GPU runtime when serving CUDA TEI images.
Installation
Production dependencies:
make install
Development environment:
make dev-install
Equivalent uv command:
uv sync --group dev --all-extras
Configuration
Embedder example:
kind: embedder
model:
model-id: /models/testb
dtype: float32
pooling: cls
server:
hostname: 0.0.0.0
port: 8082
batching:
max-batch-tokens: 16384
max-client-batch-size: 32
Reranker example:
kind: reranker
model:
model-id: BAAI/bge-reranker-base
dtype: float16
server:
hostname: 0.0.0.0
port: 8081
batching:
max-batch-tokens: 16384
max-client-batch-size: 32
Running
Local CLI, assuming TEI is available on PATH:
tei-serving --config configs/embedder.yaml
Equivalent:
python -m tei_serving.main --config configs/embedder.yaml
Docker build:
docker build -t tei-serving:local .
Docker run with a mounted config:
docker run --gpus all --rm \
-v /absolute/path/to/config.yaml:/config/config.yaml:ro \
-v /absolute/path/to/models:/models:ro \
tei-serving:local \
--config /config/config.yaml
Model Serving
The runner passes the configured model-id directly to TEI for both embedder and reranker configs. It does not copy local model directories, download Hugging Face snapshots, or rewrite SentenceTransformers metadata before startup.
Development
make format
make lint
make type-check
make security
make test
make test-cov
make ci
Repository Layout
src/tei_serving/
__init__.py # runner implementation
exceptions.py # package exceptions
main.py # CLI entrypoint
settings.py # Pydantic settings and CLI serialization
configs/
embedder.yaml # example embedder config
reranker.yaml # example reranker config
tests/
unit/ # unit tests for settings and runner behavior
Metadata
Release files for tei-serving 1.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| tei_serving-1.2.0.tar.gz | 4.1 kB | Without provenance | Details for tei_serving-1.2.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| tei_serving-1.2.0-py3-none-any.whl | Python 3 | none | any | Details for tei_serving-1.2.0-py3-none-any.whl |
Total release size: 9.6 kB
Release files / tei_serving-1.2.0.tar.gz
| Download URL | tei_serving-1.2.0.tar.gz |
|---|---|
| Size | 4.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c7e794ef159884845cca49a362b358d997ccb3dbd7f596590cd66d1c939f8eb7
|
|
BLAKE2b-256 checksum How to use checksums |
963327a816a976154eea189cc4b6acaaaf42458704b13ad444f854f00b175d5f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.19 {"installer":{"name":"uv","version":"0.11.19","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"12","id":"bookworm","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / tei_serving-1.2.0-py3-none-any.whl
| Download URL | tei_serving-1.2.0-py3-none-any.whl |
|---|---|
| Size | 5.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
6116ba026f7feb85e515cbcd477e4c8a9e38e63da27225dd36407e773ac4a2d0
|
|
BLAKE2b-256 checksum How to use checksums |
65cc5f136141aaeaec4ec0470fe82b0d80485835f5e2c93ea29905c9b0197414
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.19 {"installer":{"name":"uv","version":"0.11.19","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"12","id":"bookworm","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for tei-serving 1.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.