tritony - Tiny configuration for Triton Inference Server

What is this?

If you see the official example, it is really confusing to use where to start.

Use tritony! You will get really short lines of code like example below.

import argparse
import os
from glob import glob
import numpy as np
from PIL import Image

from tritony import InferenceClient


def preprocess(img, dtype=np.float32, h=224, w=224, scaling="INCEPTION"):
    sample_img = img.convert("RGB")

    resized_img = sample_img.resize((w, h), Image.Resampling.BILINEAR)
    resized = np.array(resized_img)
    if resized.ndim == 2:
        resized = resized[:, :, np.newaxis]

    scaled = (resized / 127.5) - 1
    ordered = np.transpose(scaled, (2, 0, 1))
    
    return ordered.astype(dtype)


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--image_folder", type=str, help="Input folder.")
    FLAGS = parser.parse_args()

    client = InferenceClient.create_with("densenet_onnx", "0.0.0.0:8001", input_dims=3, protocol="grpc")
    client.output_kwargs = {"class_count": 1}

    image_data = []
    for filename in glob(os.path.join(FLAGS.image_folder, "*")):
        image_data.append(preprocess(Image.open(filename)))

    result = client(np.asarray(image_data))

    for output in result:
        max_value, arg_max, class_name = output[0].decode("utf-8").split(":")
        print(f"{max_value} ({arg_max}) = {class_name}")

Release Notes

• 26.10.01 v0.0.21: Clean up async tasks on cancellation; prefer TRITON_ASYNC_TASKS over ASYNC_TASKS.
• 24.07.11 Upgrade minimum tritonclient version to 2.34.0
• 23.08.30 Support optional with model input, parameters on config.pbtxt
• 23.06.16 Support tritonclient>=2.34.0
• Loosely modified the requirements related to tritonclient

Key Features

• Simple configuration. Only $host:$port and $model_name are required.
• Generating asynchronous requests with asyncio.Queue
• Simple Model switching
• Support async tritonclient

Requirements

$ pip install tritonclient[all]

Install

$ pip install tritony

Configuration

The default async task count uses TRITON_ASYNC_TASKS, then the legacy ASYNC_TASKS, then 4.

Test

With Triton

./bin/run_triton_tritony_sample.sh

pytest -s --cov-report term-missing --cov=tritony tests/

Example with image_client.py

• Follow steps in the official triton server documentation

# Download Images from https://github.com/triton-inference-server/server.git
python ./example/image_client.py --image_folder "./server/qa/images"