Open Inference Protocol gRPC Client

open-inference-grpc is a generated client library based on the gRPC protocol definition tracked in the open-inference/open-inference-protocol/ repository.

Installation

This package requires Python 3.8 or greater.

Install with your favorite tool from pypi.org/project/open-inference-grpc/

$ pip install open-inference-grpc
$ poetry add open-inference-grpc

A REST-based python client (open-inference-openapi) also exists for the Open Inference Protocol, and can be installed alongside this gRPC client, as both are distributed as namespace packages.

Example

# These dependencies are installed by open-inference-grpc
import grpc
from google.protobuf.json_format import MessageToDict

from open_inference.grpc.service import GRPCInferenceServiceStub
from open_inference.grpc.protocol import (
    ServerReadyRequest,
    ModelReadyRequest,
    ModelMetadataRequest,
    ModelInferRequest,
)


with grpc.insecure_channel("localhost:8081") as channel:
    client = GRPCInferenceServiceStub(channel)

    # Check that the server is live, and it has the iris model loaded
    client.ServerReady(ServerReadyRequest())
    client.ModelReady(ModelReadyRequest(name="iris-model"))

    # Make an inference request
    pred = client.ModelInfer(
        ModelInferRequest(
            model_name="iris-model",
            inputs=[
                {
                    "name": "input-0",
                    "datatype": "FP64",
                    "shape": [1, 4],
                    "contents": {"fp64_contents": [5.3, 3.7, 1.5, 0.2]},
                }
            ],
        )
    )

print(MessageToDict(pred))
# {
#     "modelName": "iris-model",
#     "parameters": {"content_type": {"stringParam": "np"}},
#     "outputs": [
#         {
#             "name": "output-1",
#             "datatype": "INT64",
#             "shape": ["1", "1"],
#             "parameters": {"content_type": {"stringParam": "np"}},
#             "contents": {"int64Contents": ["0"]},
#         }
#     ],
# }

async with grpc.aio.insecure_channel('localhost:8081') as channel:
    stub = GRPCInferenceServiceStub(channel)
    await stub.ServerReady(ServerReadyRequest())

Dependencies

The open-inference-grpc python package relies only on grpcio, the underlying transport implementation of gRPC.

Contribute

This client is largely generated automatically by grpc-tools, with a small amount of build post-processing in build.py.

Run python build.py to build this package, it will:

1. If proto/open_inference_grpc.proto is not found, download it from open-inference/open-inference-protocol/
2. Run grpcio_tools.protoc to create the python client
3. Postprocess filenames and imports
4. Prepend the Apache 2.0 License preamble
5. Format with black

If you want to contribute to the open-inference-protocol itself, please create an issue or PR in the open-inference/open-inference-protocol repository.

License

By contributing to Open Inference Protocol Python client repository, you agree that your contributions will be licensed under its Apache 2.0 License.