Openvino runtime for MLServer
Project description
Overview
This package provides a MLServer runtime compatible with Openvino. This package has couple features:
- If server detect that model file is onnx format script will auto convert to openvino format (xml, bin) with dynamic batch size for openvino.
- Openvino dynamic batch size
- Grpc Ready
- V2 Inference Protocol
- Models metrics
Why MLserver?
For serving Openvino I choose MLServer because this framework has V2 Inference Protocol (https://kserve.github.io/website/modelserving/inference_api/), grpc and metrics out of the box.
Install
pip install mlserver mlserver-openvino
Content Types
If no content type is present on the request or metadata, the Openvino runtime will try to decode the payload as a NumPy Array. To avoid this, either send a different content type explicitly, or define the correct one as part of your model’s metadata.
Models repository
Your models add to models folder. Accepted files: ["model.xml", "model.onnx"]
/example
/models/your-model-name/
/tests
setup.py
README.md
Training and serve example: https://mlserver.readthedocs.io/en/latest/examples/sklearn/README.html
Metrics
For download metrics (prometheus) use below links
GET http://<your-endpoint>/metrics
GET http://0.0.0.0:8080/metrics
Start docker server
# Build docker image
mlserver build . -t test
# Start server and pass mlserevr_models_dir
docker run -it --rm -e MLSERVER_MODELS_DIR=/opt/mlserver/models/ -p 8080:8080 -p 8081:8081 test
Example queries:
For example script see below files:
/example/grpc-example.py
/example/rest-example.py
Kserve usage
- First create one time kserve runtime from file: kserve/cluster-runtime.yaml
- Create InferenceService from template:
apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
name: "my-openvino-model"
spec:
predictor:
model:
modelFormat:
name: openvino
runtime: kserve-mlserver-openvino
#storageUri: "gs://kfserving-examples/models/xgboost/iris"
storageUri: https://github.com/myrepo/models/mymodel.joblib?raw=true
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for mlserver_openvino-0.4.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f305eb73bc8cc3ee5ad8e51c26380fe3148fcc041e358d0d51570e6a22d68098 |
|
MD5 | a31fa60430f191177b99925b551d9206 |
|
BLAKE2b-256 | 8f70f7fc33a03ccadc0a6a23e3a40ed1cddcec4bf795fcfa53b96ae95f6b6e19 |