benchmatrix 0.2.1

A pytest-benchmark matrix, metadata, and JSON results layer.

benchmatrix

Benchmark matrices for Python projects that need performance data they can trust, compare, and parse.

benchmatrix sits on top of pytest-benchmark and adds the layer that benchmark suites usually grow by hand: implementation-by-case matrices, strict JSON-safe metadata, metric-aware result parsing, and concise display of saved benchmark runs.

Build repeatable suites	Keep metrics honest	Parse saved runs
Generate pytest benchmark tests across implementations, cases, and metric views.	Separate latency, throughput, and local distribution comparisons instead of mixing unlike numbers.	Load benchmatrix-tagged pytest-benchmark JSON rows into structured Python objects.

Quick Start

Create a benchmark matrix from ordinary synchronous callables:

from benchmatrix import BenchmarkCase, make_benchmark_test

implementations = {
    "builtin": sum,
    "loop": lambda vs: sum(v for v in vs),
}

cases = [
    BenchmarkCase.from_values(
        "small",
        list(range(100)),
        work_units=100,
        work_unit_name="items",
    ),
]

test_sum_matrix = make_benchmark_test(implementations, cases)

Run it with pytest-benchmark and keep the machine-readable output:

uv run pytest tests/test_sum_benchmark.py --benchmark-json benchmark.json

Read the tagged rows back later:

from benchmatrix import display_benchmark_rows, load_benchmark_json

rows = load_benchmark_json("benchmark.json")
display_benchmark_rows(rows)

Why It Exists

pytest-benchmark owns timing, calibration, statistics, terminal reporting, and JSON export. benchmatrix owns the repeatable structure around those timings.

Need	benchmatrix gives you
Compare multiple implementations	One generated pytest benchmark matrix instead of repeated parametrization code.
Track what each timing means	JSON-safe invocation metadata with implementation, case, and metric identity.
Report different metric views	Single-call latency, logical-work throughput, and local tail-latency summaries.
Reuse benchmark output	Parsers and display helpers for pytest-benchmark JSON produced by benchmatrix tests.

benchmatrix is intentionally narrow: it benchmarks synchronous Python callables. It is not a load-testing framework, production latency monitor, or replacement for pytest-benchmark.

Install

For local development from this repository:

make ready

To try the package from another project before the first PyPI release, install from GitHub:

uv add "benchmatrix @ git+https://github.com/ryancswallace/benchmatrix"

Documentation

Start here	Use it for
First benchmark	A complete first benchmark from test file to parsed JSON.
Create a benchmark matrix	Cases, work units, fresh inputs, and synchronous target wrappers.
Parse benchmark results	Loading and displaying benchmatrix-tagged pytest-benchmark JSON.
Performance model	What the metrics mean and what they do not prove.
Configuration and automation	Make targets, CI workflows, Docker checks, docs, and SBOM generation.

The MkDocs site builds in strict mode and generates API reference pages from the package docstrings.

Project Links

• Examples
• Contributing
• Changelog
• Security policy
• Release policy
• Code of conduct
• Citation metadata

License

benchmatrix is distributed under the MIT License.