benchloop-client 0.1.9

bmcli — command line tool for benchmark analysis data (metrics, trajectory, reports, artifacts).

benchloop-client (bmcli)

Command line tool for fetching and downloading benchmark analysis data (run metrics, trajectory, pre-analysis reports, task/job lists, and artifacts) by group_id / job_id.

PyPI package name is benchloop_client; the installed command is bmcli.

Install package

pip install benchloop_client

bmcli --api-key $BENCH_LOOP_API_KEY overview get -j $JOB_ID
bmcli --api-key $BENCH_LOOP_API_KEY metrics get -j $JOB_ID

# Option 2: config ~/.benchloop/config.toml by cli, and run bmcli without --api-key option:
bmcli config set --api-key $BENCHLOOP_API_KEY --host https://benchmark-analysis.alibaba-inc.com

bmcli overview get -j $JOB_ID
bmcli metrics get -j $JOB_ID

Install and run from source

# with uv (recommended)
uv sync
uv run bmcli --help

# or editable install with pip
python -m venv .venv && source .venv/bin/activate
pip install -e .
bmcli --help

Auth & config

Authentication uses a personal api key sent as the X-API-Key header.

Precedence: CLI flag > env var > ~/.benchloop/config.toml > default.

• Env: BENCHLOOP_API_KEY (or BENCHMARK_API_KEY), BENCHLOOP_HOST, BENCHLOOP_OUTPUT_DIR
• Default host (prod): https://benchmark-analysis.alibaba-inc.com
• Pre host: https://pre-benchmark-analysis.alibaba-inc.com

bmcli config set --api-key $BENCHLOOP_API_KEY --host https://benchmark-analysis.alibaba-inc.com

Usage

Two-segment bmcli <resource> <action> form. get prints a JSON array to stdout (pipeable to jq); download writes the standard file layout.

bmcli metrics get -j J123 | jq '.[0]'
bmcli metrics download -g G1 --scope all -o ./out
bmcli trajectory download -g G1,G2 -o ./out
bmcli report download -j J123 --agent all
bmcli jobs get -g G1 --all          # get supports --all (auto-paginate)
bmcli jobs download -g G1           # download is always full
bmcli artifacts download -g G1 -o ./out
bmcli pull -g G1 --with metrics,trajectory,jobs,artifacts -o ./out

IDs accept repeated flags, comma-separated values, and - to read from stdin. -g and -j are mutually exclusive in a single command.

Exit codes

code	meaning
0	all success
1	partial failure (some ids failed)
2	usage / parameter error
3	auth failure (HTTP 401)
4	network / timeout

stdout carries data only; logs and progress go to stderr. --quiet suppresses non-error logs, --verbose adds debug logs.

Release new package

uv build
twine check dist/benchloop_client-0.1.7-py3-none-any.whl
twine upload dist/benchloop_client-0.1.7-py3-none-any.whl --verbose