graphnetz 0.1.1

Statistically rigorous GNN benchmarking

Why GraphNetz

Whether you are proposing a new GNN architecture, testing a model on a new graph domain, or comparing existing methods across graph types, GraphNetz turns the usual “train, evaluate, table of accuracies” workflow into a reproducible statistical report. Instead of reporting point estimates alone, it provides confidence intervals for each result, paired model comparisons with multiple-testing correction, and rank-based summaries across datasets using critical-difference diagrams. The goal is not just to crown a leaderboard winner, but to give researchers a principled way to quantify uncertainty, compare methods fairly, and produce the exact evidence reviewers often ask for in graph-learning papers.

Most GNN benchmarks report point-estimate accuracies on a handful of citation graphs and declare a winner without confidence intervals, multiple-comparison correction, or rank aggregation across datasets. GraphNetz's default output is a structured statistical report, not a raw accuracy table:

• multi-seed Student's t confidence intervals per cell,
• Holm–Bonferroni paired t-tests (or Wilcoxon signed-rank) within each task,
• Demšar critical-difference diagrams from Friedman ranks with a Nemenyi post-hoc.

The catalogue is organised along a category × task taxonomy:

• 63 dataset loaders across 10 scientific categories
• 4 task kinds (node classification, graph classification, graph regression, link prediction)
• 5 canonical architectures (GCN, GAT, GIN, GraphSAGE, Graph Transformer) plug into every kind via a small set of task-kind adapters;

Install

uv add graphnetz
# or, in an existing environment:
pip install graphnetz

For local development:

git clone https://github.com/quant-sci/graphnetz
cd graphnetz
uv sync --group dev

GraphNetz requires Python ≥ 3.10, torch ≥ 2.6, and torch-geometric ≥ 2.6.

Quick start

from graphnetz import GCN, train_node_classification, plot_history
from graphnetz.datasets.social import cora

ds = cora("data/cora")
model = GCN(ds.num_features, 64, ds.num_classes)
history = train_node_classification(model, ds[0], epochs=200)
fig, ax = plot_history(history, title="GCN on Cora")

For a full benchmark run with the default statistical report:

from graphnetz import GAT, GCN, GraphSAGE, GraphTransformer, run_benchmark

report = run_benchmark(
    "social",
    {"GCN": GCN, "GAT": GAT, "GraphSAGE": GraphSAGE, "GraphTransformer": GraphTransformer},
    seeds=(0, 1, 2, 3, 4, 5, 6, 7, 8, 9),
    kind="node_cls",          # restrict to one task family
)
print(report.summary())       # per-(task, model) mean ± t-CI
print(report.pairwise())      # Holm-corrected paired t-tests (or Wilcoxon)
fig, _ = report.plot_critical_difference(alpha=0.05)

Task kinds

Unlabelled graphs (Netzschleuder, synthetic combinatorial, Ising lattice) enter the benchmark through link prediction on a held-out edge split, so every cell carries a real test-time metric — there is no self-supervised pretext loss in the headline report.

Dataset categories

† Requires the optional ogb extra (pip install graphnetz[ogb]). The five OGB loaders are folded into their domain categories rather than exposed as a separate ogb category, so they appear in run_benchmark(category, ...) alongside the curated built-ins.

from graphnetz.datasets.social import cora, roman_empire
from graphnetz.datasets.biology import peptides_func
from graphnetz.datasets.computing import internet_as

# Optional OGB loaders live in their domain modules (require `pip install graphnetz[ogb]`):
from graphnetz.datasets.social import ogbn_arxiv     # node_cls
from graphnetz.datasets.biology import ogbg_molhiv   # graph_cls

ds_cora = cora("data/cora")
ds_rom  = roman_empire("data/roman_empire")        # heterophilic
ds_pep  = peptides_func("data/peptides_func")      # LRGB
ds_inet = internet_as("data/internet_as")          # Netzschleuder

For arbitrary Netzschleuder networks:

from graphnetz import Netz
ds = Netz(root="data", dataset_name="urban_streets", network_name="brasilia")

Models

Node-level encoders enter every task kind through three small adapters: graph-level pooling head, dot-product link-prediction head, and the DGI self-supervised wrapper for optional unsupervised pre-training.

Custom models

from graphnetz import register_model

# 1. Decorator
@register_model(kinds="node_cls")
class MyGNN(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels): ...

# 2. Class attribute (no decorator)
class MyGNN(torch.nn.Module):
    task_kinds = {"node_cls", "graph_cls"}

# 3. Inline tuple at run-time
run_benchmark(
    "social",
    {"MyGNN": (MyGNN, "node_cls",
               lambda i, h, o: MyGNN(i, h, o, dropout=0.3))},
)

The statistical report

run_benchmark(...) returns a BenchmarkReport with the following methods:

Notebooks

Worked examples live under examples/:

• 01_benchmark.ipynb — the cross-category dashboard (multi-seed report, bootstrap CIs, custom-model integration).
• 02_knowledge.ipynb — relational link prediction on FB15k-237 / WN18-RR using the DistMult decoder.

Reproducing the paper

PYTHONPATH=src uv run python paper/experiment.py   # train + cache + figures
latexmk -pdf paper/main.tex                        # compile PDF

The script trains 5 architectures × 10 seeds across the 10 surviving categories, caches the histories under paper/_cache_*.pkl, and writes every figure (paper/figures/) and LaTeX table (paper/tables/) referenced by paper/main.tex. Total runtime on a recent laptop CPU is under 30 minutes.

Contributing

Pull requests welcome. Read CONTRIBUTING.md first — the short version is: every benchmark cell must carry a real held-out metric, every change must thread through the multi-seed pipeline, and every PR must be ruff clean.

uv run pytest
uv run ruff check

License

MIT — see LICENCE.txt.