vllm-tgis-adapter 0.0.3

vLLM adapter for a TGIS-compatible grpc server

vllm-tgis-adapter

vLLM adapter for a TGIS-compatible grpc server.

Install

vllm-tgis-adapter is available on PyPi

pip install vllm-tgis-adapter
python -m vllm_tgis_adapter

Build

python -m build
pip install dist/*whl
python -m vllm_tgis_adapter

Inference

This will start serving a grpc server on port 8033. This can be queried with grpcurl:

bash examples/inference.sh

Docker

Image available at quay.io/dtrifiro/vllm-tgis

docker pull quay.io/dtrifiro/vllm-tgis

Inference

See examples

Contributing

Set up pre-commit for linting/style/misc fixes:

pip install pre-commit
pre-commit install
# to run on all files
pre-commit run --all-files

This project uses nox to manage test automation:

pip install nox
nox --list  # list available sessions
nox -s tests-3.10 # run tests session for a specific python version
nox -s build-3.11 # build the wheel package
nox -s lint-3.11 -- --mypy # run linting with type checks