inferencebench-llm
LLM inference plugin for InferenceBench Suite. Drives requests through any inference engine and produces a signed envelope with TTFT/TPOT/throughput/ goodput-at-SLO metrics.
Status
Phase 1 ships vLLM-only support. SGLang, TensorRT-LLM, llama.cpp, MLX in Phase 2.
Install
pip install inferencebench inferencebench-llm
Quickstart
# Start a vLLM server (separately) on :8000, then:
bench run llm.inference \
--model meta-llama/Llama-4-Maverick \
--engine vllm \
--endpoint http://localhost:8000/v1 \
--concurrency 1,4,16,64 \
--duration 300
See docs/plugins/llm-inference.md for the full reference.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
inferencebench_llm-0.1.0.tar.gz
(44.1 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file inferencebench_llm-0.1.0.tar.gz.
File metadata
- Download URL: inferencebench_llm-0.1.0.tar.gz
- Upload date:
- Size: 44.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.11.14 {"installer":{"name":"uv","version":"0.11.14","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
70362890498c97156c078a2df224004298184f03ff02b5fdab20e0d191e27610
|
|
| MD5 |
bf8c677d265e37874e570b2ba9755ab2
|
|
| BLAKE2b-256 |
034f948aadfe3c74076c24954fbd8bac89f3777e4dd0dc409d73d5992058f918
|
File details
Details for the file inferencebench_llm-0.1.0-py3-none-any.whl.
File metadata
- Download URL: inferencebench_llm-0.1.0-py3-none-any.whl
- Upload date:
- Size: 35.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.11.14 {"installer":{"name":"uv","version":"0.11.14","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
aaf61922a3be47ed683f2569bed020bddba6b1f439fac5f2dfb8906249ae4c71
|
|
| MD5 |
383cad1c6731d0ec3da695623fa95840
|
|
| BLAKE2b-256 |
758edcaa539a6a8f6bacdb61607ac016790aee05b2e75fea54b31fca663df2f8
|
