benchmatrix 1.0.0

Benchmark matrices for Python projects that need performance data they can trust, compare, and parse

[!TIP] Read the benchmatrix documentation for the quickstart, usage guides, and API reference.

benchmatrix sits on top of pytest-benchmark and adds the layer that benchmark suites usually grow by hand: implementation-by-case matrices, strict JSON-safe metadata, metric-aware result parsing, and concise display of saved benchmark runs.

Quick Start

Create a benchmark matrix from ordinary synchronous callables in a pytest file:

from benchmatrix import BenchmarkCase, make_benchmark_test

implementations = {
    "builtin": sum,
    "loop": lambda vs: sum(v for v in vs),
}

cases = [
    BenchmarkCase.from_values(
        "small",
        list(range(100)),
        work_units=100,
        work_unit_name="items",
    ),
]

test_sum_matrix = make_benchmark_test(implementations, cases)

Run it with pytest-benchmark and keep the machine-readable output:

uv run pytest tests/test_sum_benchmark.py --benchmark-json benchmark.json

Read the run back and compare it with a controlled baseline:

from benchmatrix import display_benchmark_rows, load_benchmark_run

baseline = load_benchmark_run("baseline.json")
candidate = load_benchmark_run("benchmark.json")

display_benchmark_rows(candidate.rows)
comparison = baseline.compare_to(candidate)

if not comparison.passed:
    for cell in comparison.regressed:
        print(cell.implementation_name, cell.case_name, cell.metric_name)

Collect a controlled repeated-run group without managing output names by hand:

benchmatrix collect --runs 5 --output benchmark-runs -- \
    uv run pytest tests/test_sum_benchmark.py

The collection directory contains numbered pytest-benchmark JSON files and an atomic benchmatrix-manifest.json. Collection can recover without discarding its audit trail:

# Continue attempts after an interrupted process.
benchmatrix collect --resume --output benchmark-runs

# Append bounded retries while preserving every failed attempt.
benchmatrix collect --retry-failed --output benchmark-runs

The manifest command and working directory are reused, accepted files are revalidated, and retry attempts append to the audit trail instead of replacing failures.

Compare collection directories directly:

benchmatrix compare baseline-runs candidate-runs \
    --threshold 5% \
    --fail-on-regression

Existing automation can still combine individual repeated files:

benchmatrix compare baseline-1.json candidate-1.json \
    --baseline-run baseline-2.json \
    --candidate-run candidate-2.json \
    --threshold 5% \
    --fail-on-regression

Keep comparison policy under review with the benchmark suite:

[tool.benchmatrix.evidence]
minimum_runs = 3

[tool.benchmatrix.regression]
default_threshold_percent = 5.0

[tool.benchmatrix.regression.by_metric]
tail_latency = 8.0

benchmatrix compare discovers the nearest pyproject.toml. CLI policy options override their corresponding configured scalar without discarding per-metric, implementation, case, or exact-cell thresholds.

Inspect or validate that policy without running benchmarks:

benchmatrix policy show
benchmatrix policy validate --quiet

Comparison JSON is a strict, versioned decision record that can be archived and loaded independently of the original timing files:

benchmatrix compare baseline-runs candidate-runs \
    --format json > comparison.json

from benchmatrix import load_comparison_report

report = load_comparison_report("comparison.json")
print(report.schema_version, report.passed, len(report.regressed))

Publish the same report as Markdown or a GitHub Actions step summary:

benchmatrix compare baseline-runs candidate-runs --format markdown
benchmatrix compare baseline-runs candidate-runs --github-summary

Why It Exists

pytest-benchmark owns timing, calibration, statistics, terminal reporting, and JSON export. benchmatrix owns the repeatable structure around those timings.

benchmatrix is intentionally narrow: it benchmarks synchronous Python callables. It is not a load-testing framework, production latency monitor, or replacement for pytest-benchmark.

Interpreting Results

Benchmark output is environment-specific. Compare results only between runs from controlled environments, and keep the pytest-benchmark JSON output with the hardware, Python, dependency, and CI context that produced it.

Use metric names as part of every comparison:

• single-call latency compares one completed synchronous target call;
• batch throughput compares logical work per second when work_units is meaningful and consistent;
• tail-latency summaries describe local distribution shape for a benchmark run, not production service latency.

Install

Install the released package with uv:

uv add benchmatrix

or with pip:

python -m pip install benchmatrix

For local development from this repository:

make ready

Documentation

The documentation source lives under docs/. The top-level Markdown files are short project entry points; detailed guides, explanations, references, and runbooks live in the MkDocs documentation.

The MkDocs site builds in strict mode and generates API reference pages from the package docstrings.

Project Links

• Examples
• Contributing
• Changelog
• Security policy
• Release policy
• Code of conduct
• Citation metadata

License

benchmatrix is distributed under the MIT License.