Triton Inference Server Model

Simple package to run inference with Triton Inference Server easily.

pip install trism
# Or
pip install https://github.com/hieupth/trism

How to use

# Create triton model.
model = TritonModel(
  model="my_model",     # Model name.
  version=0,            # Model version.
  url="localhost:8001", # Triton Server URL.
  grpc=True             # Use gRPC or Http.
)
# View metadata.
for inp in model.inputs:
  print(f"name: {inp.name}, shape: {inp.shape}, datatype: {inp.dtype}\n")
for out in model.outputs:
  print(f"name: {out.name}, shape: {out.shape}, datatype: {out.dtype}\n")
# Inference.
outputs = model.run(data = [np.array(...)])

License

GNU AGPL v3.0.
Copyright © 2024 Hieu Pham. All rights reserved.