Skip to main content

benchloop-client (bl, bmcli)

Command line tool for fetching and downloading benchmark analysis data (run metrics, trajectory, metadata, pre-analysis reports, task/job lists, and artifacts) by group_id / job_id / download task_id, and for submitting Behavior Reward Model (brm) analysis tasks.

PyPI package name is benchloop_client; the installed commands are bl and bmcli.

Install package

pip install benchloop_client

bl --api-key $BENCH_LOOP_API_KEY overview get -j $JOB_ID
bl --api-key $BENCH_LOOP_API_KEY metrics get -j $JOB_ID

# Option 2: config ~/.benchloop/config.toml by cli, and run bl without --api-key option:
bl config set --api-key $BENCHLOOP_API_KEY --host https://benchmark-analysis.alibaba-inc.com

bl overview get -j $JOB_ID
bl metrics get -j $JOB_ID

The CLI checks PyPI for a newer release in the background at most once every 24 hours. Update notices include the current/latest versions and the upgrade command. Notices are written only to stderr, so JSON output remains pipeable.

Install and run from source

# with uv (recommended)
uv sync
uv run bl --help

# or editable install with pip
python -m venv .venv && source .venv/bin/activate
pip install -e .
bl --help

Auth & config

Authentication uses a personal api key sent as the X-API-Key header.

Precedence: CLI flag > env var > ~/.benchloop/config.toml > default.

  • Env: BENCHLOOP_API_KEY (or BENCHMARK_API_KEY), BENCHLOOP_HOST, BENCHLOOP_OUTPUT_DIR
  • Default host (prod): https://benchmark-analysis.alibaba-inc.com
  • Pre host: https://pre-benchmark-analysis.alibaba-inc.com
bl config set --api-key $BENCHLOOP_API_KEY --host https://benchmark-analysis.alibaba-inc.com

Usage

Two-segment bl <resource> <action> form. get prints a JSON array to stdout (pipeable to jq); download writes the standard file layout. Global options such as --api-key, --host, --timeout, --retry, --concurrency, --quiet, and --verbose must be placed before the resource name. Use --concurrency 1 for a per-file byte progress bar, or a higher value for concurrent requests/downloads.

bl metrics get -j J123 | jq '.[0]'
bl metrics download -g G1 --scope all -o ./out
bl --concurrency 10 trajectory download -g G1,G2 -o ./out
bl --concurrency 10 trajectory download --task-id TASK_ID -o ./out
bl metadata download --task-id TASK_ID -o ./out
bl report download -j J123 -o ./out
bl report get -g G123
bl report download -g G123 -o ./out
bl rsi get -j J123
bl rsi trigger -j J123
bl rsi download -g G123 -o ./out
bl jobs get -g G1 --all          # get supports --all (auto-paginate)
bl jobs download -g G1           # download is always full
bl artifacts download -g G1 -o ./out
bl artifacts download --task-id TASK_ID -o ./out
bl pull -g G1 --with metrics,trajectory,jobs,artifacts -o ./out
bl brm run -g G1
bl brm run -g G1 --steps 10,20 --sample-size 512 --scoring-limit 32
bl brm get --task-id UID
bl brm get --task-id UID --scope overall
bl brm get --task-id UID --scope detail
bl brm get --task-id UID --scope status
bl brm status --task-id UID
bl brm list -g G1

Metrics and Job trajectory requests always explicitly use the Trace source. The CLI does not expose a trace/artifacts source selector.

Group trajectory retrieval and downloads use the backend Trace export API. The CLI waits for the export URL, downloads the gzip JSON-lines payload, and keeps the existing group_id/trajectories JSON layout. The export wait timeout is 30 minutes by default; override it in seconds with --download-timeout SECONDS on trajectory commands or pull.

Group downloads always auto-paginate and include every Job in the Group. In particular, trajectory, detailed metrics, report, and artifacts enumerate the complete Job list instead of relying on the backend Group endpoints that return at most 1000 Jobs. Existing aggregate files keep their original paths:

out/G1/trajectory.json
out/G1/detail_metrics.json
out/G1_report.md
out/G1/J123_report.md
out/G1/J123/<artifact-name>
out/TASK_ID_trajectories.zip
out/TASK_ID_metadata.zip
out/TASK_ID/J123.tgz

report download -g writes both the Group summary report and each Job report. pull -g uses the same complete download behavior for every selected resource. Use --skip-existing to resume file-based downloads after an interruption.

IDs accept repeated flags, comma-separated values, and - to read from stdin. -g and -j are mutually exclusive in a single command. For trajectory and artifact downloads, --task-id is a third mutually exclusive input mode.

Download trajectories by task ID

The benchmark analysis Jobs page can turn the current filters into a trajectory download task. Copy the command shown by the page and run it locally:

bl trajectory download --task-id TASK_ID -o ./out

The CLI fetches the task's complete job ID list, downloads job trajectories with fixed concurrency, and prints count/ETA progress to stderr. It creates ./out/TASK_ID_trajectories.zip with this layout:

trajectories/<jobId>_trajectory.json
metadata/<jobId>.json
manifest.json

Metadata is fetched from /openapi/jobs/{job_id}/meta and archived alongside the trajectory. If trajectory download succeeds but metadata fails, the trajectory is still archived, the metadata error is recorded in manifest.metadata_failures, and the command exits with code 1. If only some jobs fail, the successful files are still archived. If every job fails, no empty ZIP is created.

Download metadata by task ID

Use metadata download when you only need per-job metadata for a download task:

bl metadata download --task-id TASK_ID -o ./out

The CLI creates ./out/TASK_ID_metadata.zip with this layout:

metadata/<jobId>.json
manifest.json

Partial metadata failures are recorded in manifest.failures; successful metadata files are still archived and the command exits with code 1.

Download artifacts

Artifacts can be downloaded by Job, Group, or download task:

bl artifacts download -j J123 -o ./out
bl artifacts download -g G1 -o ./out
bl artifacts download --task-id TASK_ID -o ./out
bl artifacts download -g G1 --include "*.tgz" --exclude "debug-*" -o ./out
bl artifacts download -g G1 --list-only

For Job and Group mode, the CLI still lists artifacts first via /openapi/jobs/{job_id}/artifacts, but binary download is routed through the backend proxy endpoint /openapi/jobs/{job_id}/artifacts/download instead of directly using OSS presigned URLs. In task mode, every job from the task snapshot is downloaded as ./out/TASK_ID/<jobId>.tgz.

When an artifact is missing, the backend may return either HTTP 404 or an HTML error page. The CLI treats both as a missing artifact and writes a valid empty .tgz placeholder instead of saving the error page, so downstream batch extraction can continue.

Artifact downloads show byte-level progress. On a TTY the progress is refreshed in place; in CI or redirected stderr it falls back to periodic [progress] lines. --quiet suppresses progress output.

Submit a Behavior Reward Model (brm) task

brm run creates a Behavior analysis task for exactly one Group. The CLI first calls GET /openapi/behavior/steps?group_id=... to detect whether the Group has step sampling, then posts POST /openapi/behavior/tasks.

# No-step groups (all_jobs sampling): omit --steps
bl brm run -g G1

# Step groups: --steps is required (comma-separated or repeatable)
bl brm run -g G1 --steps 10,20
bl brm run -g G1 --steps 10 --steps 20 --sample-size 256 --scoring-limit 64

# Pipe the task id
bl brm run -g G1 --steps 10,20 | jq '.[0].uid'

Defaults: --sample-size 512 (step mode only), --scoring-limit 32 (always sent). In step mode, omitting --steps is a usage error and the available steps are listed on stderr. Passing --steps on a no-step Group is also a usage error.

On success, stderr logs a short confirmation and stdout prints a JSON array (one object) that includes at least uid, plus group_id and samplingMode (step or all_jobs). Use .[0].uid (not necessarily task_id) for follow-up brm status / brm get calls.

Query a Behavior Reward Model (brm) task

brm get queries task status and/or analysis by Behavior task uid (from brm run). Multiple --task-id values are supported (repeatable or comma-separated); stdout is always a JSON array, one element per id, matching metrics get -j ....

# Default --scope all (status + overall + detail); samplingMode only here
bl brm get --task-id UID
bl brm get --task-id UID --scope all | jq '.[0] | {status, overall, detail, samplingMode}'

# overall: analysis.summary + categoryScores
bl brm get --task-id UID --scope overall | jq '.[0] | {summary, categoryScores}'

# detail: analysis.behaviorPatterns
bl brm get --task-id UID --scope detail | jq '.[0].behaviorPatterns'

# status only; `brm status` is equivalent
bl brm get --task-id UID --scope status
bl brm status --task-id UID
bl brm status --task-id UID1,UID2

--scope values:

scope source payload
overall GET .../analysis summary, categoryScores
detail GET .../analysis behaviorPatterns
status GET .../tasks/{uid} task status / progress
all (default) both endpoints nested status, overall, detail, plus top-level samplingMode

brm status always uses the status endpoint (same as brm get --scope status).

List Behavior Reward Model (brm) tasks by Group

brm list calls GET /openapi/behavior/tasks?group_id=... and prints a JSON array (one element per -g), matching jobs get -g ...:

bl brm list -g G1
bl brm list -g G1,G2
bl brm list -g G1 | jq '.[0].tasks[].uid'

Each element looks like { "group_id", "total", "tasks": [ ... ] }. Task objects keep the backend fields and also set task_id from uid when present.

Exit codes

code meaning
0 all success
1 partial failure (some ids failed)
2 usage / parameter error
3 auth failure (HTTP 401)
4 network / timeout

stdout carries data only; logs and progress go to stderr. --quiet suppresses non-error logs, --verbose adds debug logs.

Release new package

uv build
twine check dist/benchloop_client-*.whl
twine upload dist/benchloop_client-*.whl --verbose

Metadata

Release files for benchloop-client 0.2.25

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for benchloop-client 0.2.25
File Size Uploaded Provenance Actions
benchloop_client-0.2.25.tar.gz 98.8 kB Without provenance Details for benchloop_client-0.2.25.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for benchloop-client 0.2.25
File Interpreter ABI Platform Actions
benchloop_client-0.2.25-py3-none-any.whl Python 3 none any Details for benchloop_client-0.2.25-py3-none-any.whl

Total release size: 166.2 kB

Release files / benchloop_client-0.2.25.tar.gz

Download URL benchloop_client-0.2.25.tar.gz
Size 98.8 kB
Tags Source
SHA-256 checksum
How to use checksums
2daff586e008acaea66a3fd1746629d49bdfe03518ea2b394f62fab8ade96658
BLAKE2b-256 checksum
How to use checksums
803ea1807b1b4825373af87eee6734604b5bfcf7ec60841bfb60b5c52b1e1390
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.13

Release files / benchloop_client-0.2.25-py3-none-any.whl

Download URL benchloop_client-0.2.25-py3-none-any.whl
Size 67.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
3d2ced007c36daddf6e9caf2dd8c8d1e731cec6d79583ca78ef45868e6cd819c
BLAKE2b-256 checksum
How to use checksums
eea21ef6471401bbc4f07c50eba8f8c9a584610ac673092f1dee0e34c372e3d0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.13

Security for benchloop-client 0.2.25

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page