aiSSEMBLE™ Open Inference Protocol KServe

The Open Inference Protocol (OIP) specification defines a standard protocol for performing machine learning model inference across serving runtimes for different ML frameworks. This Python application can be leveraged to deploy KServe that are compatible with the Open Inference Protocol.

Installation

Add aissemble-open-inference-protocol-kserve to an application

pip install aissemble-open-inference-protocol-kserve

Usage

Prerequisite

In order to stand up KServe Using aiSSEMBLE Open Inference Protocol, user should make sure all infrastructure/environment for KServe is set up using the official Documentation. Once KServe environment is set up, user can proceed with implementing custom handler for KServe using aiSSEMBLE Open Inference Protocol.

Implementing a Handler

To make a custom handler to integrate with Kserve, create your class and extend the DataplaneHandler. Then, implement methods based on the model such as load and infer.

Example of Usage with a Handler

Create your custom handler class with:

from typing import Optional

from aissemble_open_inference_protocol_shared.handlers.model_handler import (
    ModelHandler,
)
from aissemble_open_inference_protocol_shared.types.dataplane import (
    InferenceRequest,
    InferenceResponse,
    ModelMetadataResponse,
    MetadataTensor,
    Datatype,
)

class MyHandler(ModelHandler):
    def __init__(self):
        super().__init__()

    def infer(
            self,
            payload: InferenceRequest,
            model_name: str,
            model_version: Optional[str] = None,
    ) -> InferenceResponse:
        return InferenceResponse(
            model_name=model_name, model_version=model_version, id="id", outputs=[]
        )

    def model_metadata(
            self,
            model_name: str,
            model_version: Optional[str] = None,
    ) -> ModelMetadataResponse:
        # Return a stub ModelMetadataResponse
        return ModelMetadataResponse(
            name=model_name,
            versions=[model_version] if model_version else None,
            platform="python",
            inputs=[MetadataTensor(name="input", datatype=Datatype.FP32, shape=[1])],
            outputs=[
                MetadataTensor(name="output", datatype=Datatype.FP32, shape=[1])
            ],
        )

    def model_load(self, model_name: str) -> bool:
        # Do some model loading
        return True

You can now use this handler to create the AissembleOIPKServe class to be loaded into the Kserve inferencing server Example Kserve inferencing server

from aissemble_open_inference_protocol_kserve.aissemble_oip_kserve import (
    AissembleOIPKServe,
)

if __name__ == "__main__":
    model_name = "my_model"
    oip_kserve = AissembleOIPKServe(name=model_name, model_handler=MyHandler())
    # load() should be called before start server.
    # which will call the handler's model_load() to ensure model is loaded
    oip_kserve.load()
    oip_kserve.start_server()

You are now ready to containerize the app and pass it to the Kserve Kubernetes resources.

Configurations

There are several configurations available that affect the server. These can be implemented via container arguments (passed through InferenceService YAML in the args field), environment variables, or Krausening properties file oip.properties.

Configuration Precedence

Configuration values are resolved in the following order of precedence (highest to lowest):

1. Container arguments (e.g., --http_port=9000 passed via InferenceService YAML args field)
2. Environment variables (e.g., KSERVE_HTTP_PORT=9000)
3. Krausening properties (e.g., kserve_http_port=9000 in oip.properties)
4. Default values (as shown in the table above)

Additional configuration options may be available via container arguments or environment variables. See the KServe documentation for more details.

Examples

For working examples, refer to the Examples documentation.