Skip to main content

AI Infra Bench

AI Infra Bench measures OpenAI-compatible inference endpoints, evaluates model outputs, and finds the highest load that satisfies a service-level objective. It is backend-independent and does not require a serving framework SDK.

Documentation

Read the documentation: 简体中文 | English.

The user documentation source covers setup, benchmarking, SLO search, session replay, evaluation, and monitoring in Simplified Chinese and English. The MDX site includes language switching and a GitHub Pages workflow. See website development and publishing to preview or deploy it.

Install

pip install ai-infra-bench

Python 3.10 or newer is required.

CLI

Send one request:

aib req --base-url http://127.0.0.1:30000 --prompt "Who are you?"

With no arguments, aib req sends the default prompt Who are you when run directly in a terminal. With piped or redirected input, it reads a complete JSON request payload from stdin:

cat payload.json | aib req

Use --payload-stdin or --payload-path - to read from stdin when passing other options. The input must be a single JSON object.

Use --prompt - (or --user-prompt -) for plain text from a pipe or file redirection. Whitespace and newlines are preserved:

cat prompt.txt | aib req --prompt - --max-tokens 128
aib req --prompt - < prompt.txt

Run a random-token benchmark:

aib bench \
  --base-url http://127.0.0.1:30000 \
  --dataset random \
  --input-len 1024 \
  --output-len 256 \
  --random-range-ratio 0.5 \
  --num-requests 100 \
  --max-concurrency 16

When --random-range-ratio is set, each random request samples its input and output length uniformly from the configured ratio of the target length up to the target length. The default value of 1.0 preserves fixed-length requests.

Run the packaged GPQA Diamond multiple-choice workload (198 questions):

aib bench \
  --base-url http://127.0.0.1:30000 \
  --dataset gpqa \
  --num-requests 198 \
  --max-concurrency 16

Evaluate the text-only portion of Humanity's Last Exam directly through a chat-completions endpoint. Local JSONL files are supported; image questions are skipped because this adapter does not require a multimodal runtime:

aib eval-dataset \
  --evals hle \
  --dataset-path ./hle.jsonl \
  --num-shots 0 \
  --num-questions 20 \
  --base-url http://127.0.0.1:30000 \
  --model your-model

Other commands cover dataset evaluation, logits and hidden-state comparison, metric plotting, and local Prometheus monitoring:

aib --help

Replay session-shaped JSONL payloads with one concurrency slot per session:

aib session-bench 'sessions/**/*.jsonl' \
  --max-concurrency 16 \
  --num-warmup-sessions 3

Requests in each file are sent in order. Different files run concurrently up to --max-concurrency.

SLO searches are configured in YAML. The command probes the configured range and returns the highest max_concurrency or request_rate that satisfies every condition:

endpoint:
  base_url: http://127.0.0.1:30000
  api_key: EMPTY
  model: null
request:
  payload:
    messages:
      - role: user
        content: hello
benchmark:
  num_requests: 100
  warmup_requests: 5
  max_concurrency: 32
  request_rate: inf
search:
  parameter: max_concurrency
  min: 1
  max: 64
conditions:
  - metric: success_rate
    operator: ">="
    value: 0.99
  - metric: p99_latency_ms
    operator: "<"
    value: 3000

Run it with:

aib slo examples/slo.yaml -o slo-result.yaml

Output

SLO runs write a YAML report with every probe, its metrics, and condition results. Request benchmarks can also write machine-readable JSON or JSONL metrics for later visualization.

Inspect model weights

pip install 'ai-infra-bench[weight]'
aib check-weight --path /path/to/model

Local safetensors inspection needs only the lightweight safetensors and ml-dtypes extras, in addition to the base package. It reads tensor metadata and scalar values without loading full weight tensors. BF16, FP8, and packed FP4 parameter accounting do not require PyTorch. Dtypes are displayed using safetensors names such as BF16, F32, and U8.

For a Hugging Face model ID, also install huggingface-hub. Legacy .bin checkpoints still require a separately installed torch; it is only imported when reading that format.

Development

python -m pip install -e .
pytest

Licensed under the Apache License 2.0.

Metadata

Release files for ai-infra-bench 0.1.4.post8

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ai-infra-bench 0.1.4.post8
File Size Uploaded Provenance Actions
ai_infra_bench-0.1.4.post8.tar.gz 93.8 kB Without provenance Details for ai_infra_bench-0.1.4.post8.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for ai-infra-bench 0.1.4.post8
File Interpreter ABI Platform Actions
ai_infra_bench-0.1.4.post8-py3-none-any.whl Python 3 none any Details for ai_infra_bench-0.1.4.post8-py3-none-any.whl

Total release size: 203.4 kB

Release files / ai_infra_bench-0.1.4.post8.tar.gz

Download URL ai_infra_bench-0.1.4.post8.tar.gz
Size 93.8 kB
Tags Source
SHA-256 checksum
How to use checksums
983e671943c9ade98ac3fb5e97558995f0d73b80cc27a37aca206429fe0ec7d7
BLAKE2b-256 checksum
How to use checksums
e1ea200274de4937dcad1750e7a32ace7d0c29a8879ab562196db8474b03a8e8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.16

Release files / ai_infra_bench-0.1.4.post8-py3-none-any.whl

Download URL ai_infra_bench-0.1.4.post8-py3-none-any.whl
Size 109.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
09efa47593a2cb54a3c43c3a832c7abf7d778ff22a9d4aa87476d77405e9dd36
BLAKE2b-256 checksum
How to use checksums
389b7715cfaa5de058971ee5b7ca6166b7e9a1c2bc83f573b6d92afc767a7e53
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.16

Security for ai-infra-bench 0.1.4.post8

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page