perfall-cli

perfall-cli is the standalone client package for uploading benchmark artifacts to Perfall. It is a lightweight client that can be installed independently of the server.

Install from PyPI:

pip install perfall-cli

Supported commands currently live under perfall ingest and cover:

• pytest-benchmark

• go-test

• google-benchmark

• jmh

• criterion

• benchmarkdotnet

• vitest

Regression gating is available on every ingest adapter via the shared flags:

perfall ingest pytest-benchmark \
  --file benchmark.json \
  --api-key "$PERFALL_API_KEY" \
  --testbed "github-actions-ubuntu-24.04-python-3.12" \
  --baseline-branch master \
  --baseline-window 5 \
  --fail-on-regression-percent 5

That upload still publishes the run, then compares each metric against the recent compatible baseline-branch samples (five by default), aggregates them with a median, and exits non-zero when the threshold is exceeded. Latency and cost metrics are lower-is-better; normalized throughput metrics such as ops/s, B/s, and items/s are higher-is-better, so a throughput drop past the threshold fails the gate. A metric warms up without failing until the selected baseline window is available; pass --baseline-window 1 to explicitly compare only the latest sample.

Use --testbed for CI gates to keep runner or container changes from comparing unlike results. Pick a stable explicit value such as github-actions-ubuntu-24.04-python-3.12; a changed testbed warms up without failing until it has enough history. Omitting --testbed preserves the legacy unscoped baseline behavior for existing workflows.

For example, gate a JMH throughput benchmark exactly as you would a latency benchmark:

perfall ingest jmh --file jmh-results.json --api-key "$PERFALL_API_KEY" \
  --baseline-branch master --fail-on-regression-percent 5

Local development:

python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -e .[testing]
pytest