pytest-park 0.2.2

Organise and analyse your pytest benchmarks

pytest-park

Organise and analyse your pytest benchmarks

Features

• Inline benchmark comparison printed directly in pytest output — no extra commands needed.
• Load pytest-benchmark JSON artifact folders and normalize runs, groups, marks, params, and custom grouping metadata.
• Compare reference runs against candidate runs with per-case and per-group delta and speedup summaries.
• Flexible grouping: custom keys, benchmark groups, marks, params, and postfix-based name normalization.
• Associate optional profiler artifacts with benchmark runs for code-level context.
• Serve an interactive local NiceGUI dashboard for historical exploration.

Installation

With pip:

python -m pip install pytest-park

With uv:

uv add --group test pytest-park

Usage

Step 1 — Enable the plugin

Add one line to your top-level conftest.py:

# tests/conftest.py
pytest_plugins = ["pytest_park.pytest_plugin"]

Step 2 — Run your tests

pytest

After the normal pytest-benchmark tables, a pytest-park summary section is printed automatically. It compares the current run against the latest saved benchmark artifact found in pytest-benchmark storage. No extra arguments are needed.

Step 3 — Save runs to build a history (optional)

# Save and keep comparing against the latest saved run automatically
pytest --benchmark-autosave

# Save with a meaningful name for a stable reference point
pytest --benchmark-save baseline

# Compare against a specific saved run
pytest --benchmark-compare=0001
pytest --benchmark-compare=8d530304

# Save a candidate and compare it against a specific baseline
pytest --benchmark-save candidate-v2 --benchmark-compare=0001

pytest-park reuses the baseline that pytest-benchmark resolves from its configured storage — it does not require a second format. --benchmark-storage is respected as usual.

VS Code Test Explorer: if the run looks like a single-shot execution (benchmark timing disabled or reduced), pytest-park prints a warning so the output is not mistaken for a real comparison.

Name normalization and grouping (optional)

If your benchmark names encode variant postfixes (e.g. test_func_orig, test_func_ref), add the pytest_benchmark_group_stats hook to group and label variants together:

# tests/conftest.py
from pytest_park.pytest_benchmark import default_pytest_benchmark_group_stats


def pytest_benchmark_group_stats(config, benchmarks, group_by):
    return default_pytest_benchmark_group_stats(
        config,
        benchmarks,
        group_by,
        original_postfix="_orig",
        reference_postfix="_ref",
        group_values_by_postfix={
            "_orig": "original",
            "_ref": "reference",
        },
    )

This stores parsed parts in extra_info["pytest_park_name_parts"] (base_name, parameters, postfix) and groups paired variants under the same row in the comparison table.

To expose the postfix options as pytest flags:

def pytest_addoption(parser):
    parser.addoption("--benchmark-original-postfix", action="store", default="")
    parser.addoption("--benchmark-reference-postfix", action="store", default="")

Custom grouping metadata (optional)

Store arbitrary metadata on a benchmark for richer grouping:

def test_compute_optimized(benchmark):
    benchmark.extra_info["custom_groups"] = {
        "technique": "vectorization",
        "scenario": "large-batch",
    }
    benchmark(compute)

Group by any key with --group-by custom:technique in the CLI.

---

CLI — deeper analysis across saved artifacts

Use the CLI when you want to compare specific saved runs, apply advanced grouping, or include profiler data.

# Compare latest run (candidate) against second-latest run (reference)
pytest-park analyze ./.benchmarks

# Compare named runs
pytest-park analyze ./.benchmarks --reference baseline --candidate candidate-v2

# When only --candidate is given, the preceding run is used as reference
pytest-park analyze ./.benchmarks --candidate candidate-v2

# Group by benchmark group and a specific parameter
pytest-park analyze ./.benchmarks --group-by group --group-by param:device

# Group by custom metadata key
pytest-park analyze ./.benchmarks --group-by custom:scenario

# Exclude a parameter from comparison
pytest-park analyze ./.benchmarks --exclude-param device

# Keep a parameter as a separate dimension
pytest-park analyze ./.benchmarks --group-by group --distinct-param device

# Normalize method names by stripping postfixes
pytest-park analyze ./.benchmarks --original-postfix _orig --reference-postfix _ref

# Include profiler artifacts
pytest-park analyze ./.benchmarks --profiler-folder ./.profiler --group-by group

# Print installed version
pytest-park version

Grouping reference

Default precedence (when no --group-by is given): custom > benchmark_group > marks > params

Token	Alias(es)	Resolves to
custom:<key>	—	extra_info["custom_groups"]["<key>"]
custom	custom_group	All custom group keys combined
group	benchmark_group	Benchmark group label
marks	mark	Comma-joined pytest marks
params	—	All parameter key=value pairs
param:<name>	—	Value of a specific parameter
name	method	Normalized method name
fullname	nodeid	Full test node path

Multiple --group-by tokens can be combined; the resulting label is joined with |.

Artifact folder expectations

• Input files are pytest-benchmark JSON files (--benchmark-save output) stored anywhere under the folder.
• Default comparison: latest run as candidate, second-latest as reference.
• When only --candidate is given, the run immediately preceding it is used as reference.
• Run identity uses metadata.run_id, metadata.tag, or fallback datetime identifiers.

---

Interactive dashboard

For exploratory, visual analysis across many saved runs:

pytest-park serve ./.benchmarks --reference baseline --host 127.0.0.1 --port 8080

# With profiler data
pytest-park serve ./.benchmarks --profiler-folder ./.profiler --port 8080

Access the dashboard at http://127.0.0.1:8080. Features include run selection, history charts, delta distribution, and method-level drill-down.

To launch a guided interactive CLI session instead:

pytest-park

---

Docs

uv run mkdocs build -f ./mkdocs.yml -d ./_build/

Update template

copier update --trust -A --vcs-ref=HEAD

Credits

This project was generated with