Skip to main content

benchloop-client (bl, bmcli)

Command line tool for fetching and downloading benchmark analysis data (run metrics, trajectory, pre-analysis reports, task/job lists, and artifacts) by group_id / job_id.

PyPI package name is benchloop_client; the installed commands are bl and bmcli.

Install package

pip install benchloop_client

bl --api-key $BENCH_LOOP_API_KEY overview get -j $JOB_ID
bl --api-key $BENCH_LOOP_API_KEY metrics get -j $JOB_ID

# Option 2: config ~/.benchloop/config.toml by cli, and run bl without --api-key option:
bl config set --api-key $BENCHLOOP_API_KEY --host https://benchmark-analysis.alibaba-inc.com

bl overview get -j $JOB_ID
bl metrics get -j $JOB_ID

The CLI checks PyPI for a newer release in the background at most once every 24 hours. Update notices include the current/latest versions and the upgrade command. Notices are written only to stderr, so JSON output remains pipeable.

Install and run from source

# with uv (recommended)
uv sync
uv run bl --help

# or editable install with pip
python -m venv .venv && source .venv/bin/activate
pip install -e .
bl --help

Auth & config

Authentication uses a personal api key sent as the X-API-Key header.

Precedence: CLI flag > env var > ~/.benchloop/config.toml > default.

  • Env: BENCHLOOP_API_KEY (or BENCHMARK_API_KEY), BENCHLOOP_HOST, BENCHLOOP_OUTPUT_DIR
  • Default host (prod): https://benchmark-analysis.alibaba-inc.com
  • Pre host: https://pre-benchmark-analysis.alibaba-inc.com
bl config set --api-key $BENCHLOOP_API_KEY --host https://benchmark-analysis.alibaba-inc.com

Usage

Two-segment bl <resource> <action> form. get prints a JSON array to stdout (pipeable to jq); download writes the standard file layout.

bl metrics get -j J123 | jq '.[0]'
bl metrics download -g G1 --scope all -o ./out
bl trajectory download -g G1,G2 -o ./out
bl trajectory download --task-id TASK_ID -o ./out
bl report download -j J123 -o ./out
bl report get -g G123
bl report download -g G123 -o ./out
bl jobs get -g G1 --all          # get supports --all (auto-paginate)
bl jobs download -g G1           # download is always full
bl artifacts download -g G1 -o ./out
bl pull -g G1 --with metrics,trajectory,jobs,artifacts -o ./out

Group downloads always auto-paginate and include every Job in the Group. In particular, trajectory, detailed metrics, report, and artifacts enumerate the complete Job list instead of relying on the backend Group endpoints that return at most 1000 Jobs. Existing aggregate files keep their original paths:

out/G1/trajectory.json
out/G1/detail_metrics.json
out/G1_report.md
out/G1/J123_report.md
out/G1/J123/<artifact-name>

report download -g writes both the Group summary report and each Job report. pull -g uses the same complete download behavior for every selected resource. Use --skip-existing to resume file-based downloads after an interruption.

IDs accept repeated flags, comma-separated values, and - to read from stdin. -g and -j are mutually exclusive in a single command. For trajectory downloads, --task-id is a third mutually exclusive input mode.

Download filtered trajectories by task ID

The benchmark analysis Jobs page can turn the current filters into a trajectory download task. Copy the command shown by the page and run it locally:

bl trajectory download --task-id TASK_ID -o ./out

The CLI fetches the task's complete job ID list, downloads job trajectories with fixed concurrency, and prints count/ETA progress to stderr. It creates ./out/TASK_ID_trajectories.zip, containing one <jobId>_trajectory.json per successful job and a manifest.json with total, success, and failure details. If only some jobs fail, the successful files are still archived and the command exits with code 1. If every job fails, no empty ZIP is created.

Exit codes

code meaning
0 all success
1 partial failure (some ids failed)
2 usage / parameter error
3 auth failure (HTTP 401)
4 network / timeout

stdout carries data only; logs and progress go to stderr. --quiet suppresses non-error logs, --verbose adds debug logs.

Release new package

uv build
twine check dist/benchloop_client-0.1.7-py3-none-any.whl
twine upload dist/benchloop_client-0.1.7-py3-none-any.whl --verbose

Metadata

Release files for benchloop-client 0.2.20

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for benchloop-client 0.2.20
File Size Uploaded Provenance Actions
benchloop_client-0.2.20.tar.gz 70.1 kB Without provenance Details for benchloop_client-0.2.20.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for benchloop-client 0.2.20
File Interpreter ABI Platform Actions
benchloop_client-0.2.20-py3-none-any.whl Python 3 none any Details for benchloop_client-0.2.20-py3-none-any.whl

Total release size: 118.1 kB

Release files / benchloop_client-0.2.20.tar.gz

Download URL benchloop_client-0.2.20.tar.gz
Size 70.1 kB
Tags Source
SHA-256 checksum
How to use checksums
d43ec3ecae5d79fbb9670ea712cf9337a93cd2091c390c65b132e753606bf9c4
BLAKE2b-256 checksum
How to use checksums
b8d8fcb91b74c37e497a2198917ceb98cf145523373049b1710849cf9049d766
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.10

Release files / benchloop_client-0.2.20-py3-none-any.whl

Download URL benchloop_client-0.2.20-py3-none-any.whl
Size 48.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
56266e6295bc17db13f2177aee3df40c7fd4faa0b1def25a5dd797f8ec86deef
BLAKE2b-256 checksum
How to use checksums
1c687ae138ad3c4c1d5cdfb75cab41e5794b5aa7159a829d6bfd954a9443c460
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.10

Security for benchloop-client 0.2.20

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page