perf-guard 0.2.0

Performance regression monitor for ML inference projects

perf-guard

perf-guard is a zero-intrusion command-line tool that turns the result directories of your existing benchmark / eval scripts into a versioned performance history. Record snapshots, compare against one or many baselines, and plot long-term trends.

Highlights

• Zero intrusion. Does not run your benchmarks; only parses the output.
• Config-driven. All metric extraction is declared in perf_guard.yaml.
• Multi-baseline compare. Compare the current run against baseline, last_week, main, or any tag in one command.
• Trend charts. Render the whole .perf_history/ as an interactive HTML chart, or print ASCII sparklines in the terminal.
• CI-ready. Exits with code 1 on regression so pipelines fail automatically.

Install

pip install perf-guard

Quick start

1. Drop a perf_guard.yaml at the project root:

   results_dir: ./results
   
   metrics:
     - name: total_eval_time
       file: summary.txt
       pattern: "Eval time:\\s+([\\d.]+)s"
       threshold_pct: 5
       direction: lower_is_better
   
     - name: pusht_success_rate
       file: pusht/eval.log
       pattern: "success_rate=([\\d.]+)%"
       threshold_pct: 2
       direction: higher_is_better
   

2. Record your reference run and tag it:

   perf-guard record results/20260421_054042 --tag baseline
   

3. After your next run, compare and detect regressions:

   perf-guard compare                               # latest vs baseline
   perf-guard compare --base baseline --base last_week
   perf-guard compare --base main --current latest
   

Commands

Command	Purpose
perf-guard record <dir> [--tag …]	Extract metrics from a result dir and store a snapshot.
perf-guard compare [--base …] [--current …]	Compare current against one or more baselines.
perf-guard list	List all recorded snapshots.
perf-guard report <ref>	Show full metrics for one snapshot.
perf-guard trend [--ref …] [--metric …] [-o path] [--ascii]	Plot metric trends across history.
perf-guard install-hook	Install a git post-commit hook.

Ref resolution

Anywhere a <ref> is accepted, the following are valid:

• latest — the most recently recorded snapshot
• A user tag (created with record --tag)
• A dirname (e.g. 20260421_054042)
• yesterday, last_week, last_month — closest snapshot to that time
• <N>d_ago or <N>_days_ago — closest snapshot to N days ago
• latest~<N> — N records before the latest

Multi-baseline compare

perf-guard compare --base baseline --base last_week --current latest

Prints one table per baseline. Exits with 1 if any baseline shows a regression.

Trend chart

# HTML chart (Chart.js via CDN) — default
perf-guard trend -o trend.html

# Restrict to a few snapshots / metrics
perf-guard trend --ref baseline --ref dtype-fix --ref compile-opt \
                 --metric total_eval_time --metric pusht_success_rate

# ASCII sparklines in the terminal
perf-guard trend --ascii

Exit codes

Code	Meaning
0	Success, no regressions
1	Config error, ref not found, or regression detected
2	Result directory does not exist

Development

git clone https://github.com/huxie/perf-guard
cd perf-guard
pip install -e .

Build and publish to PyPI:

pip install build twine
python -m build
python -m twine upload dist/*

License

MIT