AI Infra Bench
AI Infra Bench measures OpenAI-compatible inference endpoints, evaluates model outputs, and finds the highest load that satisfies a service-level objective. It is backend-independent and does not require a serving framework SDK.
Install
pip install ai-infra-bench
Python 3.10 or newer is required.
CLI
Send one request:
aib req --base-url http://127.0.0.1:30000 --prompt "Who are you?"
Run a random-token benchmark:
aib bench \
--base-url http://127.0.0.1:30000 \
--dataset random \
--input-len 1024 \
--output-len 256 \
--num-requests 100 \
--max-concurrency 16
Other commands cover dataset evaluation, logits and hidden-state comparison, metric plotting, and local Prometheus monitoring:
aib --help
SLO Search
SLO searches are configured in YAML. The command probes the configured range
and returns the highest max_concurrency or request_rate that satisfies every
condition:
endpoint:
base_url: http://127.0.0.1:30000
api_key: EMPTY
model: null
request:
payload:
messages:
- role: user
content: hello
benchmark:
num_requests: 100
warmup_requests: 5
max_concurrency: 32
request_rate: inf
search:
parameter: max_concurrency
min: 1
max: 64
conditions:
- metric: success_rate
operator: ">="
value: 0.99
- metric: p99_latency_ms
operator: "<"
value: 3000
Run it with:
aib slo examples/slo.yaml -o slo-result.yaml
Output
SLO runs write a YAML report with every probe, its metrics, and condition results. Request benchmarks can also write machine-readable JSON or JSONL metrics for later visualization.
Development
python -m pip install -e .
pytest
Licensed under the Apache License 2.0.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file ai_infra_bench-0.1.4.post5.tar.gz.
File metadata
- Download URL: ai_infra_bench-0.1.4.post5.tar.gz
- Upload date:
- Size: 73.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.12.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
98cf22f634c99cb06b06ebd8aa627b89784086ad10d64c5770a145cc0cb1c2d5
|
|
| MD5 |
84ac1c9bd45472ac15dfa3853870d6e1
|
|
| BLAKE2b-256 |
c0911c950c59e647ed61a9c5a4d315ac36eb552bf45c8ae38e817aad042ec272
|
File details
Details for the file ai_infra_bench-0.1.4.post5-py3-none-any.whl.
File metadata
- Download URL: ai_infra_bench-0.1.4.post5-py3-none-any.whl
- Upload date:
- Size: 93.1 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.12.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
e67037f7198959758128e1215aa08d108c04cc3330bcb49ba0ac386b05ef28f4
|
|
| MD5 |
093d236bd27292fde69381ed625861a6
|
|
| BLAKE2b-256 |
326529383ba4c037118a1c31ee2673ddf5979b1b55604f913f74690852a2eddf
|
