benchloop-client 0.2.8

Benchloop CLI — command line tool for benchmark analysis data (metrics, trajectory, reports, artifacts).

benchloop-client (bl, bmcli)

Command line tool for fetching and downloading benchmark analysis data (run metrics, trajectory, pre-analysis reports, task/job lists, and artifacts) by group_id / job_id.

PyPI package name is benchloop_client; the installed commands are bl and bmcli.

Install package

pip install benchloop_client

bl --api-key $BENCH_LOOP_API_KEY overview get -j $JOB_ID
bl --api-key $BENCH_LOOP_API_KEY metrics get -j $JOB_ID

# Option 2: config ~/.benchloop/config.toml by cli, and run bl without --api-key option:
bl config set --api-key $BENCHLOOP_API_KEY --host https://benchmark-analysis.alibaba-inc.com

bl overview get -j $JOB_ID
bl metrics get -j $JOB_ID

Install and run from source

# with uv (recommended)
uv sync
uv run bl --help

# or editable install with pip
python -m venv .venv && source .venv/bin/activate
pip install -e .
bl --help

Auth & config

Authentication uses a personal api key sent as the X-API-Key header.

Precedence: CLI flag > env var > ~/.benchloop/config.toml > default.

• Env: BENCHLOOP_API_KEY (or BENCHMARK_API_KEY), BENCHLOOP_HOST, BENCHLOOP_OUTPUT_DIR
• Default host (prod): https://benchmark-analysis.alibaba-inc.com
• Pre host: https://pre-benchmark-analysis.alibaba-inc.com

bl config set --api-key $BENCHLOOP_API_KEY --host https://benchmark-analysis.alibaba-inc.com

Usage

Two-segment bl <resource> <action> form. get prints a JSON array to stdout (pipeable to jq); download writes the standard file layout.

bl metrics get -j J123 | jq '.[0]'
bl metrics download -g G1 --scope all -o ./out
bl trajectory download -g G1,G2 -o ./out
bl trajectory download --task-id TASK_ID -o ./out
bl report download -j J123 -o ./out
bl report get -g G123
bl report download -g G123 -o ./out
bl jobs get -g G1 --all          # get supports --all (auto-paginate)
bl jobs download -g G1           # download is always full
bl artifacts download -g G1 -o ./out
bl pull -g G1 --with metrics,trajectory,jobs,artifacts -o ./out

IDs accept repeated flags, comma-separated values, and - to read from stdin. -g and -j are mutually exclusive in a single command. For trajectory downloads, --task-id is a third mutually exclusive input mode.

Download filtered trajectories by task ID

The benchmark analysis Jobs page can turn the current filters into a trajectory download task. Copy the command shown by the page and run it locally:

bl trajectory download --task-id TASK_ID -o ./out

The CLI fetches the task's complete job ID list, downloads job trajectories with fixed concurrency, and prints count/ETA progress to stderr. It creates ./out/TASK_ID_trajectories.zip, containing one <jobId>_trajectory.json per successful job and a manifest.json with total, success, and failure details. If only some jobs fail, the successful files are still archived and the command exits with code 1. If every job fails, no empty ZIP is created.

Exit codes

code	meaning
0	all success
1	partial failure (some ids failed)
2	usage / parameter error
3	auth failure (HTTP 401)
4	network / timeout

stdout carries data only; logs and progress go to stderr. --quiet suppresses non-error logs, --verbose adds debug logs.

Release new package

uv build
twine check dist/benchloop_client-0.1.7-py3-none-any.whl
twine upload dist/benchloop_client-0.1.7-py3-none-any.whl --verbose