Skip to main content

Tiny configuration for Triton Inference Server

Project description

tritony - Tiny configuration for Triton Inference Server

Pypi CI Coverage Status

What is this?

If you see the official example, it is really confusing to use where to start.

Use tritony! You will get really short lines of code like example below.

import argparse
import os
from glob import glob
import numpy as np
from PIL import Image

from tritony import InferenceClient

def preprocess(img, dtype=np.float32, h=224, w=224, scaling="INCEPTION"):
    sample_img = img.convert("RGB")

    resized_img = sample_img.resize((w, h), Image.Resampling.BILINEAR)
    resized = np.array(resized_img)
    if resized.ndim == 2:
        resized = resized[:, :, np.newaxis]

    scaled = (resized / 127.5) - 1
    ordered = np.transpose(scaled, (2, 0, 1))
    return ordered.astype(dtype)

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--image_folder", type=str, help="Input folder.")
    FLAGS = parser.parse_args()

    client = InferenceClient.create_with("densenet_onnx", "", input_dims=3, protocol="grpc")
    client.output_kwargs = {"class_count": 1}

    image_data = []
    for filename in glob(os.path.join(FLAGS.image_folder, "*")):

    result = client(np.asarray(image_data))

    for output in result:
        max_value, arg_max, class_name = output[0].decode("utf-8").split(":")
        print(f"{max_value} ({arg_max}) = {class_name}")

Release Notes

  • 23.08.30 Support optional with model input, parameters on config.pbtxt
  • 23.06.16 Support tritonclient>=2.34.0
  • Loosely modified the requirements related to tritonclient

Key Features

  • Simple configuration. Only $host:$port and $model_name are required.
  • Generating asynchronous requests with asyncio.Queue
  • Simple Model switching
  • Support async tritonclient


$ pip install tritonclient[all]


$ pip install tritony


With Triton

pytest -s --cov-report term-missing --cov=tritony tests/

Example with

# Download Images from
python ./example/ --image_folder "./server/qa/images"

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tritony-0.0.16.tar.gz (12.8 kB view hashes)

Uploaded Source

Built Distribution

tritony-0.0.16-py3-none-any.whl (10.6 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page