benchmatrix 0.3.0

A pytest-benchmark matrix, metadata, and JSON results layer.

benchmatrix

Benchmark matrices for Python projects that need performance data they can trust, compare, and parse.

benchmatrix sits on top of pytest-benchmark and adds the layer that benchmark suites usually grow by hand: implementation-by-case matrices, strict JSON-safe metadata, metric-aware result parsing, and concise display of saved benchmark runs.

Build repeatable suites	Keep metrics honest	Parse saved runs
Generate pytest benchmark tests across implementations, cases, and metric views.	Separate latency, throughput, and local distribution comparisons instead of mixing unlike numbers.	Load benchmatrix-tagged pytest-benchmark JSON rows into structured Python objects.

Quick Start

Create a benchmark matrix from ordinary synchronous callables in a pytest file:

from benchmatrix import BenchmarkCase, make_benchmark_test

implementations = {
    "builtin": sum,
    "loop": lambda vs: sum(v for v in vs),
}

cases = [
    BenchmarkCase.from_values(
        "small",
        list(range(100)),
        work_units=100,
        work_unit_name="items",
    ),
]

test_sum_matrix = make_benchmark_test(implementations, cases)

Run it with pytest-benchmark and keep the machine-readable output:

uv run pytest tests/test_sum_benchmark.py --benchmark-json benchmark.json

Read the tagged rows back later:

from benchmatrix import display_benchmark_rows, load_benchmark_json

rows = load_benchmark_json("benchmark.json")
display_benchmark_rows(rows)

Why It Exists

pytest-benchmark owns timing, calibration, statistics, terminal reporting, and JSON export. benchmatrix owns the repeatable structure around those timings.

Need	benchmatrix gives you
Compare multiple implementations	One generated pytest benchmark matrix instead of repeated parametrization code.
Track what each timing means	JSON-safe invocation metadata with implementation, case, and metric identity.
Report different metric views	Single-call latency, logical-work throughput, and local tail-latency summaries.
Reuse benchmark output	Parsers and display helpers for pytest-benchmark JSON produced by benchmatrix tests.

benchmatrix is intentionally narrow: it benchmarks synchronous Python callables. It is not a load-testing framework, production latency monitor, or replacement for pytest-benchmark.

Interpreting Results

Benchmark output is environment-specific. Compare results only between runs from controlled environments, and keep the pytest-benchmark JSON output with the hardware, Python, dependency, and CI context that produced it.

Use metric names as part of every comparison:

• single-call latency compares one completed synchronous target call;
• batch throughput compares logical work per second when work_units is meaningful and consistent;
• tail-latency summaries describe local distribution shape for a benchmark run, not production service latency.

Install

Install the released package with uv:

uv add benchmatrix

or with pip:

python -m pip install benchmatrix

For local development from this repository:

make ready

Documentation

The documentation source lives under docs/. The top-level Markdown files are short project entry points; detailed guides, explanations, references, and runbooks live in the MkDocs documentation.

Start here	Use it for
First benchmark	A complete first benchmark from test file to parsed JSON.
Create a benchmark matrix	Cases, work units, fresh inputs, and synchronous target wrappers.
Parse benchmark results	Loading and displaying benchmatrix-tagged pytest-benchmark JSON.
Performance model	What the metrics mean and what they do not prove.
Development	Local setup, test commands, and repository layout.
Compatibility	Supported Python versions, API stability, and support policy.
Publishing	Release artifacts, draft releases, PyPI publishing, and verification.
Configuration and automation	Make targets, CI workflows, Docker checks, docs, and SBOM generation.

The MkDocs site builds in strict mode and generates API reference pages from the package docstrings.

Project Links

• Examples
• Contributing
• Changelog
• Security policy
• Release policy
• Code of conduct
• Citation metadata

License

benchmatrix is distributed under the MIT License.