tritonclient-futures

特性

• 基于标准库concurrent与requests封装.
• 完整复现tritonclient，上手简单.

安装

pip install tritonclient-futures

示例

from tritonclient_futures import InferenceServerClient, InferInput, InferRequestedOutput, as_completed

# 客户端初始化
## url: tritonserver http url
## concurrency: 线程数
## pool_connections: 连接池个数
## pool_maxsize: 连接池的连接个数
client = InferenceServerClient(url='127.0.0.1:8000', concurrency=100, pool_connections=100, pool_maxsize=200)

# 同步调用
client.infer(model_name=model_name, inputs=inputs, outputs=outputs)

# 异步调用
fs = []
for i in iters:
    fs.append(client.async_infer(model_name=i["model_name"], inputs=i["inputs"], outputs=i["outputs"]))

resps = [f.get_result() for f in as_completed(fs)]

性能