Openvino runtime for MLServer
Project description
Overview
This package provides a MLServer runtime compatible with Openvino. This package has couple features:
- If server detect that model file is onnx format script will auto convert to openvino format (xml, bin) with dynamic batch size for openvino.
- Openvino dynamic batch size
- Grpc Ready
- V2 Inference Protocol
- Models metrics
Why MLserver?
For serving Openvino I choose MLServer because this framework has V2 Inference Protocol (https://kserve.github.io/website/modelserving/inference_api/), grpc and metrics out of the box.
Install
pip install mlserver mlserver-openvino
Content Types
If no content type is present on the request or metadata, the Openvino runtime will try to decode the payload as a NumPy Array. To avoid this, either send a different content type explicitly, or define the correct one as part of your model’s metadata.
Models repository
Your models add to models folder. Accepted files: ["model.xml", "model.onnx"]
/example
/models/your-model-name/
/tests
setup.py
README.md
Training and serve example: https://mlserver.readthedocs.io/en/latest/examples/sklearn/README.html
Metrics
For download metrics (prometheus) use below links
GET http://<your-endpoint>/metrics
GET http://0.0.0.0:8080/metrics
Start docker server
# Build docker image
mlserver build . -t test
# Start server and pass mlserevr_models_dir
docker run -it --rm -e MLSERVER_MODELS_DIR=/opt/mlserver/models/ -p 8080:8080 -p 8081:8081 test
Example queries:
For example script see below files:
/example/grpc-example.py
/example/rest-example.py
Kserve usage
- First create one time kserve runtime from file: kserve/cluster-runtime.yaml
- Create InferenceService from template:
apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
name: "my-openvino-model"
spec:
predictor:
model:
modelFormat:
name: openvino
runtime: kserve-mlserver-openvino
#storageUri: "gs://kfserving-examples/models/xgboost/iris"
storageUri: https://github.com/myrepo/models/mymodel.joblib?raw=true
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for mlserver_openvino-0.4.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 08e110427e26b982a7be9c37a375676202116acbae3666e69f5c0e7d68faa7e7 |
|
MD5 | cdb1f2501a4002f84b2264d8518680d5 |
|
BLAKE2b-256 | ac116e7b09def12f674cc34e3a51e883ec611b71a39406e741badbf846fe5511 |