gwmock-benchmark

Shared benchmarking for the gwmock packages: one harness, the committed results, and the rendered report site — in one place.

📊 Results: https://leuven-gravity-institute.github.io/gwmock-benchmark/

It exists so that every gwmock package is benchmarked the same way, so anyone can contribute results from their own hardware with a single pull request, and so the package repositories stay free of benchmark tooling, data, and figures.

How it works

1. You run a benchmark with the CLI; it writes one metrics-only JSON record.
2. You open a pull request adding that file under data/<package>/<suite>/.
3. CI validates it; on merge, the docs pipeline regenerates the figures and tables from the data — contributors never commit images or edit the docs.

Records are tiny (~1–2 KB), carry full provenance (CPU/GPU model, library versions — no hostnames), and are capped at 16 KB so the dataset stays light in the repository.

Data API

The committed records are published as static JSON at stable URLs, so you can retrieve the full dataset programmatically instead of scraping the figures and tables. Start at the manifest and follow its links:

https://leuven-gravity-institute.github.io/gwmock-benchmark/data/v1/index.json     # manifest: versions, counts, links
https://leuven-gravity-institute.github.io/gwmock-benchmark/data/v1/records.json   # every record, one request
.../data/v1/<package>/<suite>.json                                                 # e.g. signal/performance.json
.../data/v1/schema/record-v1.json                                                  # JSON Schema for a record

See the Data API page for the endpoints, the record schema, and curl/Python examples.

Benchmarked packages

More packages (gwmock-pop, gwmock-noise, gwmock) plug in as suites under src/gwmock_benchmark/suites/.

Install

uv pip install "gwmock-benchmark[signal]"   # extra per package you want to benchmark

Linux and macOS (the harness uses Unix facilities and the suites pull HPC packages such as lalsuite); Python 3.12–3.14.

CLI

# Run a benchmark (writes a record)
gwmock-benchmark signal performance --backend ripple --method batched --n-events 5000 -o out.json
gwmock-benchmark signal consistency -o data/signal/consistency

# Generate a cluster submission script for any command
gwmock-benchmark submit slurm --command "gwmock-benchmark signal performance ..." \
    --cpus 8 --gpus 1 --time 04:00:00 -o run.slurm

# Render the committed dataset into the docs (run by the docs pipeline)
gwmock-benchmark aggregate --data-dir data --docs-dir docs

Contributing results

Benchmarks from more hardware make the reports better. See the Contribute a benchmark guide for the full workflow (run locally or on a cluster → add a data file → open a PR).

Development

git clone git@github.com:Leuven-Gravity-Institute/gwmock-benchmark.git
cd gwmock-benchmark
uv venv --python 3.12 && source .venv/bin/activate
uv sync --group dev --group docs
uv run prek install
uv run pytest

Build the site locally (renders figures from data/ first):

uv run --group docs gwmock-benchmark aggregate
uv run --group docs zensical serve

License

GPL-3.0-or-later — see LICENSE.

gwmock-benchmark is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

gwmock-benchmark is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with gwmock-benchmark. If not, see https://www.gnu.org/licenses/.