benchloop-client (bl, bmcli)
Command line tool for fetching and downloading benchmark analysis data
(run metrics, trajectory, metadata, pre-analysis reports, task/job lists,
and artifacts) by group_id / job_id / download task_id, and for
submitting Behavior Reward Model (brm) analysis tasks.
PyPI package name is
benchloop_client; the installed commands areblandbmcli.
Install package
pip install benchloop_client
bl --api-key $BENCH_LOOP_API_KEY overview get -j $JOB_ID
bl --api-key $BENCH_LOOP_API_KEY metrics get -j $JOB_ID
# Option 2: config ~/.benchloop/config.toml by cli, and run bl without --api-key option:
bl config set --api-key $BENCHLOOP_API_KEY --host https://benchmark-analysis.alibaba-inc.com
bl overview get -j $JOB_ID
bl metrics get -j $JOB_ID
The CLI checks PyPI for a newer release in the background at most once every 24 hours. Update notices include the current/latest versions and the upgrade command. Notices are written only to stderr, so JSON output remains pipeable.
Install and run from source
# with uv (recommended)
uv sync
uv run bl --help
# or editable install with pip
python -m venv .venv && source .venv/bin/activate
pip install -e .
bl --help
Auth & config
Authentication uses a personal api key sent as the X-API-Key header.
Precedence: CLI flag > env var > ~/.benchloop/config.toml > default.
- Env:
BENCHLOOP_API_KEY(orBENCHMARK_API_KEY),BENCHLOOP_HOST,BENCHLOOP_OUTPUT_DIR - Default host (prod):
https://benchmark-analysis.alibaba-inc.com - Pre host:
https://pre-benchmark-analysis.alibaba-inc.com
bl config set --api-key $BENCHLOOP_API_KEY --host https://benchmark-analysis.alibaba-inc.com
Usage
Two-segment bl <resource> <action> form. get prints a JSON array to
stdout (pipeable to jq); download writes the standard file layout.
Global options such as --api-key, --host, --timeout, --retry,
--concurrency, --quiet, and --verbose must be placed before the resource
name. Use --concurrency 1 for a per-file byte progress bar, or a higher value
for concurrent requests/downloads.
bl metrics get -j J123 | jq '.[0]'
bl metrics download -g G1 --scope all -o ./out
bl --concurrency 10 trajectory download -g G1,G2 -o ./out
bl --concurrency 10 trajectory download --task-id TASK_ID -o ./out
bl metadata download --task-id TASK_ID -o ./out
bl report download -j J123 -o ./out
bl report get -g G123
bl report download -g G123 -o ./out
bl jobs get -g G1 --all # get supports --all (auto-paginate)
bl jobs download -g G1 # download is always full
bl artifacts download -g G1 -o ./out
bl artifacts download --task-id TASK_ID -o ./out
bl pull -g G1 --with metrics,trajectory,jobs,artifacts -o ./out
bl brm run -g G1
bl brm run -g G1 --steps 10,20 --sample-size 512 --scoring-limit 32
bl brm get --task-id UID
bl brm get --task-id UID --scope overall
bl brm get --task-id UID --scope detail
bl brm get --task-id UID --scope status
bl brm status --task-id UID
bl brm list -g G1
Group downloads always auto-paginate and include every Job in the Group. In
particular, trajectory, detailed metrics, report, and artifacts enumerate
the complete Job list instead of relying on the backend Group endpoints that
return at most 1000 Jobs. Existing aggregate files keep their original paths:
out/G1/trajectory.json
out/G1/detail_metrics.json
out/G1_report.md
out/G1/J123_report.md
out/G1/J123/<artifact-name>
out/TASK_ID_trajectories.zip
out/TASK_ID_metadata.zip
out/TASK_ID/J123.tgz
report download -g writes both the Group summary report and each Job report.
pull -g uses the same complete download behavior for every selected resource.
Use --skip-existing to resume file-based downloads after an interruption.
IDs accept repeated flags, comma-separated values, and - to read from stdin.
-g and -j are mutually exclusive in a single command. For trajectory and
artifact downloads, --task-id is a third mutually exclusive input mode.
Download trajectories by task ID
The benchmark analysis Jobs page can turn the current filters into a trajectory download task. Copy the command shown by the page and run it locally:
bl trajectory download --task-id TASK_ID -o ./out
The CLI fetches the task's complete job ID list, downloads job trajectories with
fixed concurrency, and prints count/ETA progress to stderr. It creates
./out/TASK_ID_trajectories.zip with this layout:
trajectories/<jobId>_trajectory.json
metadata/<jobId>.json
manifest.json
Metadata is fetched from /openapi/jobs/{job_id}/meta and archived alongside
the trajectory. If trajectory download succeeds but metadata fails, the
trajectory is still archived, the metadata error is recorded in
manifest.metadata_failures, and the command exits with code 1. If only some
jobs fail, the successful files are still archived. If every job fails, no empty
ZIP is created.
Download metadata by task ID
Use metadata download when you only need per-job metadata for a download task:
bl metadata download --task-id TASK_ID -o ./out
The CLI creates ./out/TASK_ID_metadata.zip with this layout:
metadata/<jobId>.json
manifest.json
Partial metadata failures are recorded in manifest.failures; successful
metadata files are still archived and the command exits with code 1.
Download artifacts
Artifacts can be downloaded by Job, Group, or download task:
bl artifacts download -j J123 -o ./out
bl artifacts download -g G1 -o ./out
bl artifacts download --task-id TASK_ID -o ./out
bl artifacts download -g G1 --include "*.tgz" --exclude "debug-*" -o ./out
bl artifacts download -g G1 --list-only
For Job and Group mode, the CLI still lists artifacts first via
/openapi/jobs/{job_id}/artifacts, but binary download is routed through the
backend proxy endpoint /openapi/jobs/{job_id}/artifacts/download instead of
directly using OSS presigned URLs. In task mode, every job from the task
snapshot is downloaded as ./out/TASK_ID/<jobId>.tgz.
When an artifact is missing, the backend may return either HTTP 404 or an HTML
error page. The CLI treats both as a missing artifact and writes a valid empty
.tgz placeholder instead of saving the error page, so downstream batch
extraction can continue.
Artifact downloads show byte-level progress. On a TTY the progress is refreshed
in place; in CI or redirected stderr it falls back to periodic [progress]
lines. --quiet suppresses progress output.
Submit a Behavior Reward Model (brm) task
brm run creates a Behavior analysis task for exactly one Group. The CLI first
calls GET /openapi/behavior/steps?group_id=... to detect whether the Group
has step sampling, then posts POST /openapi/behavior/tasks.
# No-step groups (all_jobs sampling): omit --steps
bl brm run -g G1
# Step groups: --steps is required (comma-separated or repeatable)
bl brm run -g G1 --steps 10,20
bl brm run -g G1 --steps 10 --steps 20 --sample-size 256 --scoring-limit 64
# Pipe the task id
bl brm run -g G1 --steps 10,20 | jq '.[0].uid'
Defaults: --sample-size 512 (step mode only), --scoring-limit 32 (always
sent). In step mode, omitting --steps is a usage error and the available
steps are listed on stderr. Passing --steps on a no-step Group is also a
usage error.
On success, stderr logs a short confirmation and stdout prints a JSON array
(one object) that includes at least uid, plus group_id and
samplingMode (step or all_jobs). Use .[0].uid (not necessarily
task_id) for follow-up brm status / brm get calls.
Query a Behavior Reward Model (brm) task
brm get queries task status and/or analysis by Behavior task uid (from
brm run). Multiple --task-id values are supported (repeatable or
comma-separated); stdout is always a JSON array, one element per id, matching
metrics get -j ....
# Default --scope all (status + overall + detail); samplingMode only here
bl brm get --task-id UID
bl brm get --task-id UID --scope all | jq '.[0] | {status, overall, detail, samplingMode}'
# overall: analysis.summary + categoryScores
bl brm get --task-id UID --scope overall | jq '.[0] | {summary, categoryScores}'
# detail: analysis.behaviorPatterns
bl brm get --task-id UID --scope detail | jq '.[0].behaviorPatterns'
# status only; `brm status` is equivalent
bl brm get --task-id UID --scope status
bl brm status --task-id UID
bl brm status --task-id UID1,UID2
--scope values:
| scope | source | payload |
|---|---|---|
overall |
GET .../analysis |
summary, categoryScores |
detail |
GET .../analysis |
behaviorPatterns |
status |
GET .../tasks/{uid} |
task status / progress |
all (default) |
both endpoints | nested status, overall, detail, plus top-level samplingMode |
brm status always uses the status endpoint (same as brm get --scope status).
List Behavior Reward Model (brm) tasks by Group
brm list calls GET /openapi/behavior/tasks?group_id=... and prints a JSON
array (one element per -g), matching jobs get -g ...:
bl brm list -g G1
bl brm list -g G1,G2
bl brm list -g G1 | jq '.[0].tasks[].uid'
Each element looks like { "group_id", "total", "tasks": [ ... ] }. Task
objects keep the backend fields and also set task_id from uid when present.
Exit codes
| code | meaning |
|---|---|
| 0 | all success |
| 1 | partial failure (some ids failed) |
| 2 | usage / parameter error |
| 3 | auth failure (HTTP 401) |
| 4 | network / timeout |
stdout carries data only; logs and progress go to stderr. --quiet suppresses
non-error logs, --verbose adds debug logs.
Release new package
uv build
twine check dist/benchloop_client-*.whl
twine upload dist/benchloop_client-*.whl --verbose
Metadata
Release files for benchloop-client 0.2.23
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| benchloop_client-0.2.23.tar.gz | 90.4 kB | Without provenance | Details for benchloop_client-0.2.23.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| benchloop_client-0.2.23-py3-none-any.whl | Python 3 | none | any | Details for benchloop_client-0.2.23-py3-none-any.whl |
Total release size: 151.7 kB
Release files / benchloop_client-0.2.23.tar.gz
| Download URL | benchloop_client-0.2.23.tar.gz |
|---|---|
| Size | 90.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2e5ab9f231c02d4957e37cf2111da4e11e891b1bd1db863c5cd30e2df13e6f7a
|
|
BLAKE2b-256 checksum How to use checksums |
1ae34bcfe3981f996f17622aa6963d256a35ea1e33116f1ec68fb13633b1c284
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.8
|
Release files / benchloop_client-0.2.23-py3-none-any.whl
| Download URL | benchloop_client-0.2.23-py3-none-any.whl |
|---|---|
| Size | 61.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
969272c1299598b6f73bac0660ac85997bc3aaab091208d8b0aba951d3bcf0eb
|
|
BLAKE2b-256 checksum How to use checksums |
48d99a333a5211bea741e660a33bd519276d6242dcfac69c60a6f2d6bd85e766
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.8
|
Security for benchloop-client 0.2.23
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.