Skip to main content

benchloop-client (bl, bmcli)

Command line tool for fetching and downloading benchmark analysis data (run metrics, trajectory, pre-analysis reports, task/job lists, and artifacts) by group_id / job_id.

PyPI package name is benchloop_client; the installed commands are bl and bmcli.

Install package

pip install benchloop_client

bl --api-key $BENCH_LOOP_API_KEY overview get -j $JOB_ID
bl --api-key $BENCH_LOOP_API_KEY metrics get -j $JOB_ID

# Option 2: config ~/.benchloop/config.toml by cli, and run bl without --api-key option:
bl config set --api-key $BENCHLOOP_API_KEY --host https://benchmark-analysis.alibaba-inc.com

bl overview get -j $JOB_ID
bl metrics get -j $JOB_ID

Install and run from source

# with uv (recommended)
uv sync
uv run bl --help

# or editable install with pip
python -m venv .venv && source .venv/bin/activate
pip install -e .
bl --help

Auth & config

Authentication uses a personal api key sent as the X-API-Key header.

Precedence: CLI flag > env var > ~/.benchloop/config.toml > default.

  • Env: BENCHLOOP_API_KEY (or BENCHMARK_API_KEY), BENCHLOOP_HOST, BENCHLOOP_OUTPUT_DIR
  • Default host (prod): https://benchmark-analysis.alibaba-inc.com
  • Pre host: https://pre-benchmark-analysis.alibaba-inc.com
bl config set --api-key $BENCHLOOP_API_KEY --host https://benchmark-analysis.alibaba-inc.com

Usage

Two-segment bl <resource> <action> form. get prints a JSON array to stdout (pipeable to jq); download writes the standard file layout.

bl metrics get -j J123 | jq '.[0]'
bl metrics download -g G1 --scope all -o ./out
bl trajectory download -g G1,G2 -o ./out
bl trajectory download --task-id TASK_ID -o ./out
bl report download -j J123 -o ./out
bl report get -g G123
bl report download -g G123 -o ./out
bl jobs get -g G1 --all          # get supports --all (auto-paginate)
bl jobs download -g G1           # download is always full
bl artifacts download -g G1 -o ./out
bl pull -g G1 --with metrics,trajectory,jobs,artifacts -o ./out

IDs accept repeated flags, comma-separated values, and - to read from stdin. -g and -j are mutually exclusive in a single command. For trajectory downloads, --task-id is a third mutually exclusive input mode.

Download filtered trajectories by task ID

The benchmark analysis Jobs page can turn the current filters into a trajectory download task. Copy the command shown by the page and run it locally:

bl trajectory download --task-id TASK_ID -o ./out

The CLI fetches the task's complete job ID list, downloads job trajectories with fixed concurrency, and prints count/ETA progress to stderr. It creates ./out/TASK_ID_trajectories.zip, containing one <jobId>_trajectory.json per successful job and a manifest.json with total, success, and failure details. If only some jobs fail, the successful files are still archived and the command exits with code 1. If every job fails, no empty ZIP is created.

Exit codes

code meaning
0 all success
1 partial failure (some ids failed)
2 usage / parameter error
3 auth failure (HTTP 401)
4 network / timeout

stdout carries data only; logs and progress go to stderr. --quiet suppresses non-error logs, --verbose adds debug logs.

Release new package

uv build
twine check dist/benchloop_client-0.1.7-py3-none-any.whl
twine upload dist/benchloop_client-0.1.7-py3-none-any.whl --verbose

Metadata

Release files for benchloop-client 0.2.14

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distribution (wheel)

Table of built distributions (wheels) for benchloop-client 0.2.14
File Interpreter ABI Platform Actions
benchloop_client-0.2.14-py3-none-any.whl Python 3 none any Details for benchloop_client-0.2.14-py3-none-any.whl

Release files / benchloop_client-0.2.14-py3-none-any.whl

Download URL benchloop_client-0.2.14-py3-none-any.whl
Size 36.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
a82ea6c6bf412829c5249886ebd13982a8613b2abde9180b00e5bca8dfa0767f
BLAKE2b-256 checksum
How to use checksums
2e35591b338e880dc20df60c761ed1226c7a4687ef02dbc38e53df0c9ceb9a75
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.11.8

Security for benchloop-client 0.2.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page