AI Infra Bench
AI Infra Bench measures OpenAI-compatible inference endpoints, evaluates model outputs, and finds the highest load that satisfies a service-level objective. It is backend-independent and does not require a serving framework SDK.
Documentation
Read the documentation: 简体中文 | English.
The user documentation source covers setup, benchmarking, SLO search, session replay, evaluation, and monitoring in Simplified Chinese and English. The MDX site includes language switching and a GitHub Pages workflow. See website development and publishing to preview or deploy it.
Install
pip install ai-infra-bench
Python 3.10 or newer is required.
CLI
Send one request:
aib req --base-url http://127.0.0.1:30000 --prompt "Who are you?"
With no arguments, aib req sends the default prompt Who are you when run
directly in a terminal. With piped or redirected input, it reads a complete JSON
request payload from stdin:
cat payload.json | aib req
Use --payload-stdin or --payload-path - to read from stdin when passing other
options. The input must be a single JSON object.
Use --prompt - (or --user-prompt -) for plain text from a pipe or file
redirection. Whitespace and newlines are preserved:
cat prompt.txt | aib req --prompt - --max-tokens 128
aib req --prompt - < prompt.txt
Run a random-token benchmark:
aib bench \
--base-url http://127.0.0.1:30000 \
--dataset random \
--input-len 1024 \
--output-len 256 \
--random-range-ratio 0.5 \
--num-requests 100 \
--max-concurrency 16
When --random-range-ratio is set, each random request samples its input and
output length uniformly from the configured ratio of the target length up to
the target length. The default value of 1.0 preserves fixed-length requests.
Run the packaged GPQA Diamond multiple-choice workload (198 questions):
aib bench \
--base-url http://127.0.0.1:30000 \
--dataset gpqa \
--num-requests 198 \
--max-concurrency 16
Evaluate the text-only portion of Humanity's Last Exam directly through a chat-completions endpoint. Local JSONL files are supported; image questions are skipped because this adapter does not require a multimodal runtime:
aib eval-dataset \
--evals hle \
--dataset-path ./hle.jsonl \
--num-shots 0 \
--num-questions 20 \
--base-url http://127.0.0.1:30000 \
--model your-model
Other commands cover dataset evaluation, logits and hidden-state comparison, metric plotting, and local Prometheus monitoring:
aib --help
Replay session-shaped JSONL payloads with one concurrency slot per session:
aib session-bench 'sessions/**/*.jsonl' \
--max-concurrency 16 \
--num-warmup-sessions 3
Requests in each file are sent in order. Different files run concurrently up
to --max-concurrency.
SLO Search
SLO searches are configured in YAML. The command probes the configured range
and returns the highest max_concurrency or request_rate that satisfies every
condition:
endpoint:
base_url: http://127.0.0.1:30000
api_key: EMPTY
model: null
request:
payload:
messages:
- role: user
content: hello
benchmark:
num_requests: 100
warmup_requests: 5
max_concurrency: 32
request_rate: inf
search:
parameter: max_concurrency
min: 1
max: 64
conditions:
- metric: success_rate
operator: ">="
value: 0.99
- metric: p99_latency_ms
operator: "<"
value: 3000
Run it with:
aib slo examples/slo.yaml -o slo-result.yaml
Output
SLO runs write a YAML report with every probe, its metrics, and condition results. Request benchmarks can also write machine-readable JSON or JSONL metrics for later visualization.
Inspect model weights
pip install 'ai-infra-bench[weight]'
aib check-weight --path /path/to/model
Local safetensors inspection needs only the lightweight safetensors and
ml-dtypes extras, in addition to the base package. It reads tensor metadata and
scalar values without loading full weight tensors. BF16, FP8, and packed FP4
parameter accounting do not require PyTorch. Dtypes are displayed using
safetensors names such as BF16, F32, and U8.
For a Hugging Face model ID, also install huggingface-hub. Legacy .bin
checkpoints still require a separately installed torch; it is only imported
when reading that format.
Development
python -m pip install -e .
pytest
Licensed under the Apache License 2.0.
Metadata
Release files for ai-infra-bench 0.1.4.post8
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| ai_infra_bench-0.1.4.post8.tar.gz | 93.8 kB | Without provenance | Details for ai_infra_bench-0.1.4.post8.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| ai_infra_bench-0.1.4.post8-py3-none-any.whl | Python 3 | none | any | Details for ai_infra_bench-0.1.4.post8-py3-none-any.whl |
Total release size: 203.4 kB
Release files / ai_infra_bench-0.1.4.post8.tar.gz
| Download URL | ai_infra_bench-0.1.4.post8.tar.gz |
|---|---|
| Size | 93.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
983e671943c9ade98ac3fb5e97558995f0d73b80cc27a37aca206429fe0ec7d7
|
|
BLAKE2b-256 checksum How to use checksums |
e1ea200274de4937dcad1750e7a32ace7d0c29a8879ab562196db8474b03a8e8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.16
|
Release files / ai_infra_bench-0.1.4.post8-py3-none-any.whl
| Download URL | ai_infra_bench-0.1.4.post8-py3-none-any.whl |
|---|---|
| Size | 109.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
09efa47593a2cb54a3c43c3a832c7abf7d778ff22a9d4aa87476d77405e9dd36
|
|
BLAKE2b-256 checksum How to use checksums |
389b7715cfaa5de058971ee5b7ca6166b7e9a1c2bc83f573b6d92afc767a7e53
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.16
|
Security for ai-infra-bench 0.1.4.post8
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.