benchbro 0.5.0

Benchmark library and CLI

benchbro

benchbro is a Python benchmarking library and CLI with pytest-style discovery and rich terminal output.

Quick start

Install dependencies:

uv sync --group dev

Create benchmark cases in any importable module:

from benchbro import Case, system


@system(scope="module")
def salt() -> bytes:
    return b"-fixture"

case = Case(name="hashing", case_type="cpu", metric_type="time", tags=["fast", "core"])


@case.input()
def payload() -> bytes:
    return b"benchbro"


@case.benchmark()
def sha1(payload: bytes, salt: bytes) -> str:
    import hashlib

    return hashlib.sha1(payload + salt).hexdigest()


@case.benchmark()
def sha256(payload: bytes, salt: bytes) -> str:
    import hashlib

    return hashlib.sha256(payload + salt).hexdigest()

Standalone systems behave like lightweight fixtures:

• declare them with @system() outside any Case
• inject them into benchmarks and other systems by parameter name
• choose scope="function" (default) or scope="module"
• systems may be sync, async, yield-based, or async-yield-based
• keep them separate from inputs: inputs cannot depend on systems, and systems cannot depend on inputs

Subprocess benchmarks use a dedicated subprocess API:

import sys

from benchbro import Case, CommandSpec

case = Case(name="cli", metric_type="time")


@case.subprocess()
def python_cli() -> CommandSpec:
    return CommandSpec(argv=[sys.executable, "-c", "pass"])

• @case.subprocess() benchmarks are time-only in v1.
• Commands are explicit argv, not shell strings.
• Unexpected exit codes and timeouts fail the run.
• Long-lived subprocesses should be started in @system(...) and consumed by benchmarks or subprocess benchmarks.

Regression thresholds default to 50.0 percent warning and 100.0 percent error at the case level, and can be overridden per benchmark:

case = Case(name="hashing", warning_threshold_pct=5.0, regression_threshold_pct=10.0)

@case.benchmark(warning_threshold_pct=2.0, regression_threshold_pct=3.0)
def critical_path(payload: bytes) -> str:
    ...

Comparison metric can be configured at case-level and benchmark-level:

case = Case(name="hashing", comparison_metric="p95_s")

@case.benchmark(comparison_metric="median_s")
def critical_path(payload: bytes) -> str:
    ...

Valid comparison metrics:

• time: median_s (default), mean_s, iqr_s, p95_s, stddev_s, ops_per_sec
• memory: peak_alloc_bytes (default), net_alloc_bytes, peak_alloc_bytes_max

GC is disabled during measured iterations by default. To keep the interpreter GC behavior unchanged, set:

Case(name="hashing", gc_control="inherit")

Run benchmarks:

uv run benchbro --repeats 10 --warmup 2

When no target is provided, benchbro discovers benchmarks from:

• benchmarks/**/*.py (relative to repo root)

You can configure discovery in pyproject.toml:

[tool.benchbro.ini_options]
benchmark_paths = ["benchmarks"]
file_pattern = ["bench_*.py", "*_bench.py", "*benchmark.py", "*benchmarks.py"]

• benchmark_paths: directories to scan when no CLI target is provided
• file_pattern: glob pattern(s) for benchmark file names in directory discovery

benchbro compares against the baseline by default (.benchbro/baseline.local.json). If the baseline is missing, benchbro creates it automatically. If new cases/benchmarks are introduced later, missing entries are merged into baseline. Pass --new-baseline to replace the entire baseline with the current run. Pass --ci to use .benchbro/baseline.ci.json for baseline read/write/compare. Pass --no-compare to skip comparison while still backfilling missing benchmark entries in baseline.

By default, regular runs do not write artifacts. Use explicit output flags (--output-json, --output-csv, --output-md) when needed.

The baseline is always written to:

• .benchbro/baseline.local.json (default local mode)
• .benchbro/baseline.ci.json when using --ci

Recommended:

• ignore .benchbro/ for machine-local benchmarking artifacts.
• commit .benchbro/baseline.ci.json for CI comparisons.

Recommended .gitignore

# Benchbro local artifacts
.benchbro/*
!.benchbro/baseline.ci.json

If requested, markdown output can also be written with --output-md.

JSON artifacts include environment metadata for reproducibility (Python/runtime/platform/CPU fields) both at run level and on each benchmark entry.

CLI basics

Run selected cases/tags and write outputs:

uv run benchbro my_benchmarks.py \
  --case hashing \
  --tag fast \
  --output-json artifacts/current.json \
  --output-csv artifacts/current.csv \
  --output-md artifacts/current.md

Compare against baseline:

uv run benchbro my_benchmarks.py

Render time benchmark histograms in terminal output:

uv run benchbro my_benchmarks.py --histogram

Skip comparison for a run while still maintaining baseline structure:

uv run benchbro my_benchmarks.py --no-compare

Regression status uses each benchmark's effective thresholds (benchmark override -> case threshold -> defaults):

• warning default: 50%
• error threshold default: 100%

The comparison table shows warning and threshold values for each row.

Histograms are terminal-only in v1 and are shown for time benchmarks.

End-to-end example

For a complete runnable workflow (baseline + candidate comparison), use:

• examples/README.md
• make examples