vLLM adapter for a TGIS-compatible grpc server
Project description
vllm-tgis-adapter
vLLM adapter for a TGIS-compatible grpc server.
Install
vllm-tgis-adapter is available on PyPi
pip install vllm-tgis-adapter
python -m vllm_tgis_adapter
HealthCheck CLI
Installing the adapter also install a grpc healthcheck cli that can be used to monitor the status of the grpc server:
$ grpc_healtheck
health check...status: SERVING
See usage with
grpc_healthcheck --help
Build
python -m build
pip install dist/*whl
python -m vllm_tgis_adapter
Inference
This will start serving a grpc server on port 8033. This can be queried with grpcurl:
bash examples/inference.sh
Docker
Image available at quay.io/opendatahub/vllm, built from opendatahub-io/vllm's Dockerfile.ubi
docker pull quay.io/opendatahub/vllm
Inference
See examples
Contributing
Set up pre-commit
for linting/style/misc fixes:
pip install pre-commit
pre-commit install
# to run on all files
pre-commit run --all-files
This project uses nox
to manage test automation:
pip install nox
nox --list # list available sessions
nox -s tests-3.10 # run tests session for a specific python version
nox -s build-3.11 # build the wheel package
nox -s lint-3.11 -- --mypy # run linting with type checks
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
vllm_tgis_adapter-0.1.1.tar.gz
(44.3 kB
view hashes)
Built Distribution
Close
Hashes for vllm_tgis_adapter-0.1.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fa01db3a03d73d9cbc24527ad116cf868549fdac403f4f9126c4597f9fed65ac |
|
MD5 | 6670e477ae8e2868dbd3d74dd9dabd27 |
|
BLAKE2b-256 | 17c2b73f820feaa3fbe4dbb9fba63c97b181bc8c20f1b5ccd74836dd62b1cdb9 |