PyBench — precise microbenchmarks for Python

Measure small, focused snippets with minimal boilerplate, auto-discovery, smart calibration, and a clean CLI (pybench).

Run benchmarks with one command:

pybench run examples/ [-k keyword] [-P key=value ...]

✨ Highlights

• Simple API: use the @bench(...) decorator or suites with Bench + BenchContext.start()/end() to isolate the hot path.
• Auto-discovery: pybench run <dir> expands to **/*bench.py.
• Powerful parameterization: generate Cartesian products with params={...} or define per-case args/kwargs.
• On-the-fly overrides: -P key=value adjusts n, repeat, warmup, group, or custom params without editing code.
• Solid timing model: monotonic clock, warmup, GC control, and context fast-paths.
• Smart calibration: per-variant iteration tuning to hit a target budget.
• Rich reports: aligned tables with percentiles, iter/s, min…max, baseline markers, and speedups vs. base.
• HTML charts: export benchmarks as self-contained Chart.js dashboards with --export chart.
• History tooling: runs auto-save to .pybenchx/; list, inspect stats, clean, or compare with --vs {name,last}.

🚀 Quickstart

📦 Install

• pip

  pip install pybenchx
  

• uv

  uv pip install pybenchx
  

🧪 Example benchmark

See examples/strings_bench.py for both styles:

from pybench import bench, Bench, BenchContext

@bench(name="join", n=1000, repeat=10)
def join(sep: str = ","):
    sep.join(str(i) for i in range(100))

suite = Bench("strings")

@suite.bench(name="join-baseline", baseline=True)
def join_baseline(b: BenchContext):
    s = ",".join(str(i) for i in range(50))
    b.start(); _ = ",".join([s] * 5); b.end()

🏎️ Running

• Run all examples

  pybench run examples/
  

• Filter by name

  pybench run examples/ -k join
  

• Override params at runtime

  pybench run examples/ -P repeat=5 -P n=10000
  

🎛️ Key CLI options

• Disable color

  pybench run examples/ --no-color
  

• Sorting

  pybench run examples/ --sort time --desc
  

• Time budget per variant (calibration)

  pybench run examples/ --budget 300ms     # total per variant; split across repeats
  pybench run examples/ --max-n 1000000    # cap calibrated n
  

• Profiles

  pybench run examples/ --profile thorough  # ~1s budget, repeat=30
  pybench run examples/ --profile smoke     # no calibration, repeat=3 (default)
  

• Save / Compare / Export

  pybench run examples/ --save latest
  pybench run examples/ --save-baseline main
  pybench run examples/ --compare main --fail-on mean:7%,p99:12%
  pybench run examples/ --export chart        # HTML dashboard (Chart.js)
  pybench run examples/ --export json         # JSON next to auto-saved run
  

🗂️ Manage history & baselines

• List everything under .pybenchx/

  pybench list
  pybench list --baselines
  

• Storage stats & cleanup

  pybench stats
  pybench clean --keep 50
  

• Compare quickly

  pybench run examples/ --vs main        # named baseline
  pybench run examples/ --vs last        # last auto-saved run
  

📊 Output

Header includes CPU, Python, perf_counter clock info, total time, and profile. Table shows speed vs baseline with percent:

(pybench) $ pybench run examples/
cpu: x86_64
runtime: python 3.13.5 (x86_64-linux) | perf_counter: res=1.0e-09s, mono=True
time: 23.378s | profile: smoke, budget=-, max-n=1000000, sequential
benchmark                          time (avg)       iter/s              (min … max)          p75          p99         p995      vs base
join                                 13.06 µs       76.6 K      13.00 µs … 13.21 µs     13.08 µs     13.20 µs     13.21 µs            -
join_param[n=100,sep='-']            13.17 µs       75.9 K      12.79 µs … 13.72 µs     13.37 µs     13.70 µs     13.71 µs            -
join_param[n=100,sep=':']            13.06 µs       76.6 K      12.85 µs … 13.23 µs     13.14 µs     13.23 µs     13.23 µs            -
join_param[n=1000,sep='-']          131.75 µs        7.6 K    129.32 µs … 134.82 µs    132.23 µs    134.70 µs    134.76 µs            -
join_param[n=1000,sep=':']          135.62 µs        7.4 K    131.17 µs … 147.50 µs    136.68 µs    146.92 µs    147.21 µs            -
group: strings                                                                                                                  
join-baseline  ★                    376.07 ns        2.7 M    371.95 ns … 384.09 ns    378.96 ns    383.66 ns    383.87 ns     baseline
join-basic                          377.90 ns        2.6 M    365.89 ns … 382.65 ns    381.15 ns    382.55 ns    382.60 ns       ≈ same
concat                               10.62 µs       94.1 K      10.54 µs … 10.71 µs     10.65 µs     10.70 µs     10.71 µs 28.25× slower

💡 Tips

• Use BenchContext.start()/end() para isolar a seção crítica e evitar ruído de setup.
• Prefira --profile smoke durante o desenvolvimento; troque para --profile thorough antes de publicar números.
• Para logs, use --no-color.