tritony 0.0.9

Tiny configuration for Triton Inference Server

tritony - Tiny configuration for Triton Inference Server

What is this?

If you see the official example, it is really confusing to use where to start.

Use tritony! You will get really short lines of code like example below.

import argparse
import os
from glob import glob
import numpy as np
from PIL import Image

from tritony import InferenceClient


def preprocess(img, dtype=np.float32, h=224, w=224, scaling="INCEPTION"):
    sample_img = img.convert("RGB")

    resized_img = sample_img.resize((w, h), Image.Resampling.BILINEAR)
    resized = np.array(resized_img)
    if resized.ndim == 2:
        resized = resized[:, :, np.newaxis]

    scaled = (resized / 127.5) - 1
    ordered = np.transpose(scaled, (2, 0, 1))
    
    return ordered.astype(dtype)


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--image_folder", type=str, help="Input folder.")
    FLAGS = parser.parse_args()

    client = InferenceClient.create_with("densenet_onnx", "0.0.0.0:8001", input_dims=3, protocol="grpc")
    client.output_kwargs = {"class_count": 1}

    image_data = []
    for filename in glob(os.path.join(FLAGS.image_folder, "*")):
        image_data.append(preprocess(Image.open(filename)))

    result = client(np.asarray(image_data))

    for output in result:
        max_value, arg_max, class_name = output[0].decode("utf-8").split(":")
        print(f"{max_value} ({arg_max}) = {class_name}")

Key Features

• Simple configuration. Only $host:$port and $model_name are required.
• Generating asynchronous requests with asyncio.Queue
• Simple Model switching
• Support async tritonclient

Requirements

$ pip install tritonclient[all]

Install

$ pip install tritony

Test

With Triton

docker run --rm \
    -v ${PWD}:/models \
    nvcr.io/nvidia/tritonserver:22.01-pyt-python-py3 \
    tritonserver --model-repo=/models

pytest -m -s tests/test_tritony.py

Example with image_client.py

• Follow steps in the official triton server documentation

# Download Images from https://github.com/triton-inference-server/server.git
python ./example/image_client.py --image_folder "./server/qa/images"