kozmoserver 0.1.0

KozmoServer

KozmoServer

An inference server for machine learning models.

Overview

KozmoServer aims to provide an easy way to start serving your machine learning models through a REST and gRPC interface, fully compliant with KFServing's V2 Dataplane spec. Watch a quick video introducing the project here.

• Multi-model serving, letting users run multiple models within the same process.
• Ability to run inference in parallel for vertical scaling across multiple models through a pool of inference workers.
• Support for adaptive batching, to group inference requests together on the fly.
• Scalability with deployment in Kubernetes native frameworks, including Kozmo Core and KServe (formerly known as KFServing), where KozmoServer is the core Python inference server used to serve machine learning models.
• Support for the standard V2 Inference Protocol on both the gRPC and REST flavours, which has been standardised and adopted by various model serving frameworks.

You can read more about the goals of this project on the initial design document.

Usage

You can install the kozmoserver package running:

pip install kozmoserver

Note that to use any of the optional inference runtimes, you'll need to install the relevant package. For example, to serve a scikit-learn model, you would need to install the kozmoserver-sklearn package:

pip install kozmoserver-sklearn

For further information on how to use KozmoServer, you can check any of the available examples.

Inference Runtimes

Inference runtimes allow you to define how your model should be used within KozmoServer. You can think of them as the backend glue between KozmoServer and your machine learning framework of choice. You can read more about inference runtimes in their documentation page.

Out of the box, KozmoServer comes with a set of pre-packaged runtimes which let you interact with a subset of common frameworks. This allows you to start serving models saved in these frameworks straight away. However, it's also possible to write custom runtimes.

Out of the box, KozmoServer provides support for:

Framework	Supported	Documentation
Scikit-Learn	✅	KozmoServer SKLearn
XGBoost	✅	KozmoServer XGBoost
Spark MLlib	✅	KozmoServer MLlib
LightGBM	✅	KozmoServer LightGBM
CatBoost	✅	KozmoServer CatBoost
Tempo	✅	github.com/kozmoai/tempo
MLflow	✅	KozmoServer MLflow
Supervisor-Detect	✅	KozmoServer Supervisor Detect
Supervisor-Explain	✅	KozmoServer Supervisor Explain
HuggingFace	✅	KozmoServer HuggingFace

KozmoServer is licensed under the Apache License, Version 2.0. However please note that software used in conjunction with, or alongside, KozmoServer may be licensed under different terms. For example, Supervisor Detect and Supervisor Explain are both licensed under the Business Source License 1.1. For more information about the legal terms of products that are used in conjunction with or alongside KozmoServer, please refer to their respective documentation.

Supported Python Versions

🔴 Unsupported

🟠 Deprecated: To be removed in a future version

🟢 Supported

🔵 Untested

Python Version	Status
3.7	🔴
3.8	🔴
3.9	🟢
3.10	🟢
3.11	🔵
3.12	🔵

Examples

To see KozmoServer in action, check out our full list of examples. You can find below a few selected examples showcasing how you can leverage KozmoServer to start serving your machine learning models.

• Serving a scikit-learn model
• Serving a xgboost model
• Serving a lightgbm model
• Serving a catboost model
• Serving a tempo pipeline
• Serving a custom model
• Serving an supervisor-detect model
• Serving a HuggingFace model
• Multi-Model Serving with multiple frameworks
• Loading / unloading models from a model repository

Developer Guide

Versioning

Both the main kozmoserver package and the inference runtimes packages try to follow the same versioning schema. To bump the version across all of them, you can use the ./hack/update-version.sh script.

We generally keep the version as a placeholder for an upcoming version.

For example:

./hack/update-version.sh 0.2.0.dev1

Testing

To run all of the tests for KozmoServer and the runtimes, use:

make test

To run run tests for a single file, use something like:

tox -e py3 -- tests/batch_processing/test_rest.py