netgamut
Browse, download and cache network datasets from Python.
Seven sources are indexed, each in its own subpackage:
| Source | Datasets | Strength |
|---|---|---|
netgamut.snap |
109 | large, well-known social / web / technological networks |
netgamut.netzschleuder |
286 | domain-tagged, incl. biological, transport and multilayer |
netgamut.konect |
300 | one fixed vocabulary of format, edge type and loops; signed and temporal |
netgamut.ogb |
12 | graph-learning benchmarks, with named node categories |
netgamut.pyg |
99 | the graph-learning corpus: citation, web, social, heterophilous |
netgamut.paths |
3 | curated path data (flight itineraries, navigation trails) for memory networks |
netgamut.ahorn |
83 | higher-order data: hypergraphs, simplicial complexes, revision-pinned |
Metadata for all seven ships inside the package, so you can explore every dataset — domains, sizes, statistics, citations — without downloading anything. When you do ask for data it lands in a shared user-level cache, so a dataset is fetched once per machine no matter how many projects, notebooks or virtualenvs use it.
pip install netgamut
For a runnable tour — browsing, loading, and building memory networks from
path data — see examples/tutorial.ipynb.
Upgrading from 0.2? Generators now return network objects rather than tuples,
and some downloads load with different weights; CHANGELOG.md
has the call-by-call migration.
One domain vocabulary across every source
The sites label things differently — SNAP groups by provenance ("Autonomous systems graphs"), Netzschleuder tags by domain, AHORN by a vocabulary of its own, KONECT by kind of relationship, OGB and PyG by nothing at all — so all seven are mapped onto one vocabulary. That makes it possible to compare methods across kinds of system rather than kinds of provenance:
import netgamut
netgamut.domains()
{'Social': 366, 'Informational': 159, 'Biological': 51,
'Technological': 32, 'Transportation': 42, 'Economic': 91}
for match in netgamut.search(domain="Biological", max_nodes=500):
print(match)
# netzschleuder/blumenau_drug [Biological] (75 nodes, 181 edges)
# netzschleuder/celegans_2019 [Biological] (453 nodes, 4,879 edges)
# netzschleuder/celegans_metabolic [Biological] (453 nodes, 4,596 edges)
# ...
net = netgamut.load("celegans_metabolic")
Note Biological: 51 — 36 from Netzschleuder, 5 each from KONECT and OGB, 4
from AHORN, 1 from PyG, and none at all from SNAP. That imbalance is why the other sources exist.
The mapping is deliberately visible in
netgamut/_core/domains.py rather than buried per source, so it can be argued
with in one place.
For anything source-specific, use that source's own search():
from netgamut import netzschleuder as nz
from netgamut import snap
nz.search(multilayer=True) # 17 multiplex networks; SNAP has none
nz.search(subtype="Food web")
snap.search(ground_truth=True) # reference partitions
Look before you download (SNAP)
from netgamut import snap
for category in snap.categories():
print(category)
Social networks (24 datasets)
Networks with ground-truth communities (9 datasets)
Communication networks (4 datasets)
Citation networks (3 datasets)
Collaboration networks (5 datasets)
...
The categories are exactly SNAP's own — same names, same grouping, same order as the index page: 23 categories over 109 datasets.
SNAP cross-lists some datasets, so a dataset carries all the categories that list it and appears once overall:
snap.info("soc-RedditHyperlinks").category_names
# ('Social networks', 'Signed networks', 'Temporal networks', 'Online communities')
len(snap.datasets()) # 109 unique datasets, no repeats
len(snap.datasets("Signed networks"))
print(snap.describe("ca-GrQc"))
ca-GrQc
=======
Category Collaboration networks
Summary Collaboration network of Arxiv General Relativity
Type Undirected
Page https://snap.stanford.edu/data/ca-GrQc.html
Reported by SNAP
Type Undirected
Nodes 5,242
Edges 14,496
Dataset statistics
Nodes 5242
Edges 14496
Nodes in largest WCC 4158 (0.793)
Average clustering coefficient 0.5296
Diameter (longest shortest path) 17
...
Files
* ca-GrQc.txt.gz
(* downloaded by default)
Source
J. Leskovec, J. Kleinberg and C. Faloutsos. Graph Evolution: ...
None of that touched the network.
Find datasets that fit
# Undirected networks small enough to hold in memory
small = snap.search(directed=False, max_edges=1_000_000)
# Everything in one category
snap.datasets("Collaboration networks")
# Free-text over names and descriptions
snap.search("bitcoin")
Each result is a Dataset with .nodes, .edges, .directed, .description,
.statistics, .files and .citations.
Categories report different columns — temporal networks have Temporal Edges
and Static Edges, review datasets have Number of items — so .columns
preserves each table's own headers verbatim rather than flattening them into a
schema SNAP never used.
Download once, reuse everywhere
net = snap.load("ca-GrQc") # downloads on first use
net = snap.load("ca-GrQc") # instant, from cache
load returns a Network handle, not a parsed graph — so the same call works
for a 14k-edge collaboration network and a billion-edge social graph.
net.path # decompressed file on disk, ready for any tool
net.dataset # the metadata from above
net.header # SNAP's own descriptive comment lines
net.size_bytes
for source, target in net.edges(cast=int): # streamed, flat memory
...
for fields in net.rows(): # full records: timestamps, signs, weights
...
Conversions are lazy and optional:
graph = net.to_networkx() # pip install "netgamut[networkx]"
im = net.to_infomap() # pip install "netgamut[infomap]"
im.run()
Both default to the direction SNAP reports for the dataset.
Writing files
Every network — downloaded or generated, first-order, memory or multilayer —
writes itself into a directory and returns the path, named after the network
unless you pass name=:
net.to_pajek("out/") # PosixPath('out/ca-GrQc.net')
net.to_link_list("out/", name="grqc") # PosixPath('out/grqc.txt')
| Network | to_pajek |
to_link_list |
|---|---|---|
first-order (Network, InMemoryNetwork) |
*Vertices + *Edges/*Arcs |
source target [weight] |
StateNetwork |
*Vertices + *States + *Links |
order 1 only |
MultilayerNetwork |
*Vertices + *Intra (+ *Inter, or *Multilayer) |
— |
Integer node ids are written as they are, so a generated network's file lines up
with its planted partition; other labels are numbered from 1 and kept as Pajek
names. A link list has nowhere to put names, so it refuses non-integer labels
rather than renumbering them silently. A written file loads the same network
Infomap gets from to_infomap(), weights included. Downloads are streamed, so
writing one never holds its links in memory.
Weights, times and layers
A third column is a weight in one file, a Unix timestamp in the next and a
sign or rating in a third, so nothing here guesses. Each source declares what
the extra columns of the file it loaded hold — Netzschleuder from the file's
own header, KONECT from its fixed source target weight time layout, SNAP from
a table of its temporal and signed files — and to_infomap(), the writers and
everything below follow it:
net = snap.load("CollegeMsg")
print(net.columns) # source target time (time)
net.weighted_links # False: the timestamp is not a weight
Temporal networks expose their times as a numpy array, read once, so you can look before you bin:
t = net.times # int64 Unix seconds, in file order
t.min(), t.max()
net.time_bins("auto") # numpy.histogram_bin_edges(t, "auto")
for snapshot in net.snapshots(bins=12): # lazy: one network per bin
print(snapshot.info["start"], snapshot.info["end"], snapshot)
bins and range mean what they mean to numpy.histogram_bin_edges: a
number of equal-width bins, explicit edges, or an estimator name. Bin i is
[edges[i], edges[i+1]), the last one closed; every bin yields, empty ones
too, and a link's weight in a snapshot is how often it occurs there (summed
weights where the file has a weight column). ISO dates come back as
datetime64.
The same bins, or a layer column, make a multilayer network:
ml = net.to_multilayer(bins=12) # layer i = bin i; ml.info["bin_edges"]
ml.to_infomap(multilayer_relax_rate=0.15, multilayer_relax_limit=1) # couple adjacent bins
netgamut.load("london_transport").to_multilayer() # its layer column
snap.load("higgs-twitter").to_multilayer(layer="interaction") # any categorical column
If a source has a column wrong, or you know better, declare it yourself:
dataclasses.replace(net, columns=netgamut.Columns(time=2)).
Some entries are multi-file bundles:
net = snap.load("ego-Facebook", file="facebook.tar.gz")
directory = net.extract()
Benchmarking Infomap
Six SNAP datasets ship reference partitions, which makes them usable for scoring community detection rather than just running it:
suite = snap.search(ground_truth=True, max_edges=5_000_000)
# com-Youtube, com-DBLP, com-Amazon
for dataset in suite:
net = snap.load(dataset.name)
result = net.to_infomap(silent=True).run()
truth = snap.load_ground_truth(dataset.name, variant="top5000")
reference = [set(community) for community in truth.communities()]
print(dataset.name, result.num_top_modules, "vs", len(reference), "ground-truth")
variant="top5000" gives SNAP's 5000 highest-quality communities — the usual
reference set; "all" gives the full list. Communities overlap, so a node can
appear in several.
Because every dataset is unique in the catalog and sizes are known up front, a graded suite is just a filter — no risk of benchmarking the same network several times because SNAP happens to list it under several headings.
A domain-balanced set for publication
CI regression wants structural diversity; a paper wants domain diversity — a different axis. SNAP alone cannot supply it (no biological networks, no multilayer, transport limited to three road networks), so draw across sources:
POOL = {
"Social": [("nz", "karate"), ("nz", "football"), ("snap", "email-Eu-core")],
"Informational": [("nz", "polbooks"), ("nz", "polblogs"), ("nz", "dblp_cite")],
"Biological": [
("nz", "celegansneural"),
("nz", "celegans_metabolic"),
("nz", "foodweb_little_rock"),
("nz", "malaria_genes"),
],
"Transportation": [
("nz", "london_transport"),
("nz", "openflights"),
("nz", "us_air_traffic"),
("snap", "roadNet-PA"),
],
"Technological": [("nz", "power"), ("nz", "internet_as"), ("nz", "linux")],
"Economic": [("nz", "fao_trade"), ("nz", "at_migrations")],
}
Reference partitions exist for karate (2 factions), football (12
conferences), polbooks/polblogs (political alignment), malaria_genes
(CysPoLV/UPS), foodweb_little_rock (trophic groups) and email-Eu-core
(42 departments).
Treat those with care: node metadata is not guaranteed to be the community
structure, a point reviewers will raise. Use synthetic benchmarks with planted
communities for the quantitative claim, and this pool for generality — reported
via codelength against baselines. Neither source provides synthetic generators —
that is what netgamut.generate is for.
Netzschleuder
from netgamut import netzschleuder as nz
nz.domains()
# {'Social': 148, 'Informational': 53, 'Biological': 36,
# 'Technological': 14, 'Transportation': 13, 'Economic': 25}
print(nz.describe("celegans_metabolic"))
Every network carries the site's precomputed measures, so sizes, clustering, assortativity and diameter are all available before downloading. Filters go beyond size:
nz.search(domain="Transportation", subtype="Airport")
nz.search(multilayer=True) # 17 multiplex networks
nz.search(weighted=True, domain="Economic")
nz.search(bipartite=True)
Downloads use the site's CSV bundle, so graph-tool is not required despite being the site's native format:
net = nz.load("celegansneural")
net.path # extracted edges.csv
list(net.edges(cast=int))
net.extract() # nodes.csv and gprops.csv too
Metadata is read from a JSON API rather than scraped, so this subpackage is markedly less fragile than the SNAP one.
KONECT
KONECT describes every network in one fixed vocabulary — a format (sym,
asym, bip), an edge type (unweighted, positive for multigraphs,
posweighted, signed, ...) and structural checks — and opens every data file
with the same two codes, % asym multisigned. That makes it the one source here
that says, for every network, whether it contains loops:
from netgamut import konect
konect.search(loops=True, max_nodes=10_000) # self-links, by KONECT's own check
konect.search(signed=True, temporal=True)
konect.search(bipartite=True, domain="Economic")
print(konect.describe("chess"))
The loop check also reaches the cross-source search, netgamut.search(loops=True),
including for a Netzschleuder or SNAP network whenever KONECT holds the identical
edge list. Networks no source has checked are left out of either filter.
The catalog is KONECT's downloadable networks minus its per-language Wikimedia series — about a thousand edit, hyperlink and talk networks, one per language edition — of which only the English editions are kept.
Overlap. Netzschleuder imported a third of KONECT, and some of KONECT is
SNAP's. Such a network stays in konect.search(), but the cross-source
netgamut.search, netgamut.domains and netgamut.load use the other copy, so
nothing is counted twice. A duplicate is decided on evidence, never on name —
KONECT's twitter is Netzschleuder's twitter_social, and Netzschleuder's
twitter is KONECT's munmun_twitter_social:
konect.info("moreno_lesmis").duplicate_of # 'netzschleuder/lesmis'
konect.duplicates() # KONECT name -> the copy used elsewhere
The evidence is either Netzschleuder naming the KONECT network as its upstream, or another entry with exactly the same node and edge count. Near-copies — a different snapshot, the largest component only — stay separate networks.
Loading. The archive is kept as KONECT serves it, and the edge list is rewritten once beside it so every tool reads it the same way:
net = konect.load("brunson_southern-women")
net.path # <name>.tsv: '%' comments become '#', which Infomap requires
net.rows() # every column kept: weights, signs, timestamps
net.extract() # KONECT's README and meta files
KONECT numbers the two sides of a bipartite network independently, so the right
side is shifted past the left: ids are unique, and right-side nodes start at
info(name).left_nodes + 1. Signed and rating weights stay in the file but are
not handed to Infomap, which accepts only non-negative weights; to_infomap()
then reads the links unweighted, summing repeated ones.
Node metadata. 27 networks ship per-node attributes as ent.* files — people's
names and sexes in moreno_crime, a whole profile per pet in the petster networks,
coauthor names in pajek-erdos. No KONECT page says which, so the catalog build
lists every archive up to 100 MB (268 of 300) and records the columns each will
yield:
konect.search(node_metadata=True)
konect.info("petster-hamster").node_attributes # ('name', 'joined', 'species', ...)
konect.load("moreno_crime").nodes()["name"] # {1: 'AbelDennis', ...}
Ids are the edge list's, so in a bipartite network a right-side entity's rows land
on the shifted ids. The 32 larger archives were not listed and answer None, which
keeps them out of either filter. A refresh re-lists an archive only when its size
or date has changed; KONECT's last changed in 2017–2018.
Node metadata, in one shape
Every source stores node attributes somewhere different — a separate SNAP artifact,
a member of a Netzschleuder bundle, KONECT's ent.* files, OGB's raw/ and
mapping/ folders, a materialised PyG table. Network.nodes() reads all of them the same way:
nodes = netgamut.load("ogbn-arxiv").nodes()
nodes.columns # ('arxiv_category', 'year', 'paper')
nodes["arxiv_category"] # {0: 'arxiv cs cr', ...} decoded where named
nodes.partition("arxiv_category") # {0: 6, ...} contiguous 0..k-1, for AMI/NMI/D
nodes.categories("arxiv_category") # ('arxiv cs ai', ...) categories[i] names module i
partition and categories are two halves of one answer: the first is what you score
a detected partition against, the second is what lets you say which category a module
lines up with. They always agree —
categories(c)[partition(c)[node]] is that node's value.
The same four calls, whatever the source:
netgamut.load("email-Eu-core").nodes()["department"] # SNAP
netgamut.load("karate", source="netzschleuder").nodes()["groups"]
netgamut.load("ogbn-arxiv").nodes()["arxiv_category"] # OGB
netgamut.load("Planetoid/Cora", source="pyg").nodes()["label"]
To find datasets that have node metadata before downloading anything, ask across sources — by presence, or by column name:
netgamut.search(node_metadata=True) # 240 datasets known to carry some
netgamut.search(node_attribute="gender") # something to score a partition against
netgamut.search("karate")[0].node_attributes # ('name', 'groups')
Each source answers from its own catalog: Netzschleuder from the site's analyses,
KONECT from its archive listings, SNAP from its label files, OGB and PyG from their
class labels. Netzschleuder, KONECT and SNAP name every column. OGB names only its
extra files (year, species) and PyG none, since the rest of their column names
come with the download. Datasets a catalog cannot answer for — all of AHORN, whose
node lines live inside the dataset file — are left out of either filter.
Column names come from the source, so a figure legend can use them as they are. Values
are decoded to names where the source ships names — OGB does, PyG does not:
Data.y is an integer tensor and PyG's API has no class_names anywhere, so
categories("label") returns the codes and their meaning lives in the originating
paper. Nodes with no value are absent from partition rather than pooled into a
module of their own, which would invent a community the source never claimed.
Overlapping ground-truth communities are a different thing and stay where they were —
snap.load_ground_truth(...) and Network.communities() — because a cover is not a
column.
Node metadata is not ground truth for community structure. A partition that disagrees with an attribute is not thereby wrong; scoring against
partition(col)measures agreement with an attribute, which is a finding about the attribute as much as about the algorithm. See Peel, Larremore & Clauset, Science Advances 3, e1602548 (2017).
Graph-learning benchmarks
netgamut.ogb and netgamut.pyg index the datasets the graph-neural-network
literature is benchmarked on — which is where most empirical networks with node
labels now live.
OGB needs no extra dependency at all. Its archives are plain zips of csv.gz, and
it is the only source that ships names for its codes:
ogb.describe("ogbn-arxiv")
ogb.search(supported=True, domain="Biological")
Not everything it publishes is loadable, and the catalog says so rather than omitting
it: the 15 ogbg-* molecule collections are not indexed at all (a Network is one
graph), while heterogeneous datasets and the two stored as data.npz are listed and
refused with their reason. Node and edge counts are read out of each archive's own
num-node-list.csv.gz, so the ones too large to download stay None rather than
being guessed.
PyG catalogues 99 single-graph datasets across 38 classes; the collections of
thousands of molecules or meshes are omitted, with pyg.omitted() giving the reason
for every class left out. A further 9 are listed but marked unloadable — PyG's
constructor wants an argument this loader does not supply — so they refuse with the
reason instead of failing deep inside PyG. Browsing needs nothing installed:
pyg.search(domain="Informational", max_nodes=50_000)
pyg.describe("WikipediaNetwork/chameleon")
Loading materialises once: the first call needs pip install "netgamut[pyg]", runs PyG's own downloader, and writes edges.csv.gz and
nodes.csv.gz into the cache. Every call after that reads those files and never
imports torch — the heavy dependency is a one-time cost, and the result is a plain,
archivable network like any other. The dense feature matrix is deliberately not
written: Cora's raw PyG directory is ~16 MB, almost all of it the 1,433-dimensional
bag-of-words matrix, against ~44 KB materialised. Use pyg.from_pyg(data) when you
want a Data object you already have — including one this package does not catalogue.
PyG is not an undirected-only collection, which is easy to assume because its
docstrings never mention direction. Of the 58 measured so far, 24 are directed —
WebKB, Actor, WikipediaNetwork/chameleon, the Airports networks, PolBlogs,
EmailEUCore and the JODIE interaction graphs — against undirected Planetoid, Amazon
and the heterophilous set.
Because PyG publishes nothing about it, direction is measured rather than assumed:
--with-direction materialises each dataset and reads the answer off its
edge_index, filling in unpublished sizes at the same time. Until a dataset has been
measured its direction is None and search(directed=...) skips it. That the
measurement is worth making is clear from Cora, which is undirected through
Planetoid and directed through AttributedGraphDataset — the preprocessing
decides, not the network, so no lookup table keyed on a name could get it right.
Both catalogs are regenerated from upstream's own registry:
python scripts/regenerate_ogb_catalog.py [--with-stats]
python scripts/regenerate_pyg_catalog.py [--show-omitted]
python scripts/regenerate_pyg_catalog.py --with-direction --max-edges 3000000
Neither needs torch for the catalog itself: OGB publishes master.csv, and PyG's
statistics live in its class docstrings, which is what its documentation cheatsheet is
generated from. Only --with-direction does, because measuring means loading. Both
--with-stats and --with-direction keep whatever a previous run established for
datasets they cannot reach, so a partial pass never resets known values to unknown.
Generating benchmarks
The generate subpackage turns any network in the pool into a synthetic
benchmark with planted communities: it reads the blueprint's modular flow —
degrees, module sizes, per-module mixing — and samples fresh networks with the
same statistics, so ground truth is known while realism is kept:
import netgamut
from netgamut import generate
G = netgamut.load("polbooks").to_networkx()
partition = generate.detect(G) # Infomap, or bring your own
net = generate.benchmark(G, partition) # degree- and mixing-preserving
net = generate.weighted_benchmark(G, partition) # + flow-model weights
Every generator returns a network object with the same methods as a download —
a InMemoryNetwork, or a StateNetwork / MultilayerNetwork for the
higher-order ones — carrying the planted partition and the sampler's
diagnostics:
net.partition # {node: module}, the ground truth
net.info # diagnostics, plus "generator" and "parameters"
H = net.to_networkx() # planted module as node attribute "module"
net.to_pajek("benchmarks/") # PosixPath('benchmarks/polbooks-benchmark.net')
Or drop the blueprint entirely and generate from fitted parameters — a generalized LFR whose degree and community-size families are chosen by AIC instead of assumed to be power laws:
deg = generate.fit_families([k for _, k in G.degree()])
size = generate.fit_families([120, 80, 60, 45, 30]) # module sizes
mix = {"family": "delta", "mu": 0.15} # flat LFR-style mixing
net = generate.parametric(1000, deg, size, mix, seed=1)
Directed blueprints keep both degree sequences and, like undirected ones, their full module hierarchy; weighted blueprints need weighted samples, since a dense weighted network carries its structure in the weights rather than the topology:
net = generate.weighted_benchmark(G, partition) # topology + flow weights
Higher-order blueprints change only the sampler, never the flow model — a memory link has to match at its physical junction, a multiplex link has to stay inside its layer:
memory = generate.memory_benchmark(links, windows, partition, seed=1) # StateNetwork
multiplex = generate.multilayer_benchmark(layers, partition, seed=1) # MultilayerNetwork
Without a blueprint, overlapping_memory_benchmark plants communities that
overlap on the physical nodes and are disjoint on the states — the regime a
first-order partition cannot represent. It is Andrea Lancichinetti's trigram
sampler, seeded, and needs only the standard library:
memory = generate.overlapping_memory_benchmark(4, 50_000, seed=1) # StateNetwork, om=4, E=50000
memory.to_pajek("out/") # out/overlapping-memory-n256-om4-nc64-E50000-mu0.1-seed1.net
memory.partition # {state id: planted community}, for `infomap -d --no-infomap -c`
Every node sits in memberships communities of community_size nodes
(defaults: 256 nodes, communities of 64, mu=0.1); mu of the trigrams cross
communities by stitching two existing pairs, so mixing adds links but no
states. A seeded network has the same states, links and planted partition as
the original script run with that seed.
The flow model itself is public as well (generate.flow_matrix,
generate.ModuleTree, generate.cross_R) for callers who want the joint
distribution rather than a sample of it. Everything here needs the extras:
pip install "netgamut[generate]".
The original LFR benchmarks
generate.lfr(...) runs the Lancichinetti–Fortunato generators themselves —
the authors' C++, not a reimplementation — so the classic baseline is one call
away, in all six of its variants:
net = generate.lfr(1000, 15, 50, mu=0.2, minc=20, maxc=50, seed=1)
net = generate.lfr(1000, 15, 50, mut=0.2, muw=0.1, beta=1.5) # weighted
net = generate.lfr(1000, 15, 50, mu=0.2, directed=True) # directed
net = generate.lfr(5000, 20, 100, mu1=0.1, mu2=0.3, # two planted
minc=10, maxc=100, # levels
min_macro=200, max_macro=1000)
net = generate.lfr(1000, 20, 50, mu=0.3, layers=10, graphs=3) # multiplex
net = generate.lfr(1000, 15, 50, mu=0.2, on=20, om=2) # overlapping
net.to_pajek("benchmarks/")
# PosixPath('benchmarks/lfr-undirected-n1000-k15-maxk50-mu0.2-seed1.net')
Which program runs follows from the arguments, and so does the shape of
net.partition: {node: module} normally, {node: (macro, micro)} for the
hierarchical variant — the path convention the flow model reads —
{node: (module, ...)} for an overlapping cover, and {(node, layer): module}
for the multiplex one, which comes back as a MultilayerNetwork
(net.to_networkx(layer=3) is one layer) ready for
generate.multilayer_benchmark. Node labels are zero-based, unlike the
one-based .dat files upstream writes. The default name, and so the file name,
spells out the variant, the main parameters and the seed; pass name= to
to_pajek when a sweep varies something else.
seed alone decides the network. Left to themselves the generators seed
from the clock; the wrapper writes time_seed.dat instead, so the same seed
gives the same network byte for byte, on any machine, with the pinned upstream
revision.
The C++ is fetched rather than vendored, and compiled where it runs. That keeps one wheel valid on every platform and needs no compiler at install time; the price is a download and a build on the first call, both cached next to the datasets:
~/.cache/netgamut/lfr/src/<commit>/ the sources, fetched once
~/.cache/netgamut/lfr/bin/<key>/ one binary per variant and toolchain
The first call takes about four seconds — Xcode command line tools on macOS,
build-essential on Linux — and every call after it is milliseconds.
generate.lfr_available() reports whether the sources and a compiler are both
in place, without downloading anything.
If you would rather not have a compiler in the loop, generate.lfr_networkx(...)
keeps networkx's reimplementation of the undirected unweighted case, with
retries around its flaky convergence.
Multilayer benchmarks with MultilayerGM
generate.multilayer_gm(...) runs
MultilayerGM (Bazzi, Jeub,
Arenas, Howison & Porter): a partition of the state nodes is sampled with a
tunable dependence between layers, then every layer gets a degree-corrected
SBM. Upstream's four steps — dependency tensor, null distribution, partition,
network — are one call:
net = generate.multilayer_gm(1000, 5, p=0.9, mu=0.2, communities=10,
theta=1, k_min=3, k_max=150, t_k=-2, seed=1)
net.to_pajek("benchmarks/") # *Intra only: couple layers with Infomap's relax rate
# PosixPath('benchmarks/mlgm-multiplex-n1000-L5-p0.9-mu0.2-seed1.net')
net = generate.multilayer_gm(1000, 20, p=0.95, coupling="temporal") # layer i copies i-1
p is how likely a node keeps its community from another layer — any other
layer for coupling="multiplex", the previous one for "temporal", where it
can also be a list of layers - 1 probabilities. mu is the fraction of edges
placed ignoring communities, which is not LFR's mixing parameter: a random
edge can still land inside a community.
There is no generator object to keep around: building the dependency tensor
and null takes milliseconds, and the time goes into sampling edges. What is
worth reusing is the partition — partition=net.partition resamples only the
edges, for a mu sweep over one planted structure.
The sampling code is vendored, at a pinned upstream commit, because upstream is
not on PyPI and seeds only through global random state; the vendored copy takes
local random streams instead. The same seed reproduces upstream's network
exactly, and never touches the caller's random or numpy.random.
Which revision, and moving it
Upstream publishes no releases — it is a repository of six source trees, last
touched in June 2024 — so the pin is a commit hash plus the SHA-256 of its
archive. generate.lfr_source() shows what a run would use:
print(generate.lfr_source())
# andrealancichinetti/LFRbenchmarks@ec9a860282d8 (pinned, fetched)
# url https://github.com/.../archive/ec9a8602....tar.gz
# sha256 ad13a354c2cfbb1de299dcb13caadd5801243b9041cd0e7e89852b9aeb384f76
# sources ~/.cache/netgamut/lfr/src/ec9a8602...
# compiler Apple clang version 17.0.0
# variants undirected, weighted, directed, weighted_directed, hierarchical, multiplex
To move it, in a checkout of this repository:
python scripts/update_lfr_pin.py --check # has upstream moved? exit 1 if so
python scripts/update_lfr_pin.py # download, verify, rewrite the pin
python scripts/update_lfr_pin.py --commit <sha> # pin something specific
The script asks GitHub what the default branch points at, downloads that
archive, refuses to pin one that is missing any of the six generators, and
rewrites COMMIT and SHA256 in src/netgamut/generate/_lfr/_sources.py.
Bumping the pin is a real change rather than housekeeping: these generators are
random, so a different revision can give a different network for the same seed.
Run pytest tests/test_lfr.py after, and say so in the commit message.
To try a revision without moving the pin, point NETGAMUT_LFR_DIR at a
checkout — the wrapper builds from there and skips the download entirely.
Comparing partitions as flow models
The same flow model compares two partitions of one network: each becomes a
joint distribution over node pairs, and the generative map distance is the
Jensen–Shannon distance between them — a metric in [0, 1] that scores a
disagreement by how much flow it misroutes, not by how many labels it moves:
D = generate.gmd(G, partA, partB) # distance, 0 when flow-equivalent
S = generate.gms(G, partA, partB) # similarity, 1 - D
D, per_node = generate.gmd(G, partA, partB, element=True) # D^2 = sum_i p_i d_i^2
generate.agmd / generate.agms are the adjusted pair, a Steinhaus transform
against the one-module null partition (generate.null_partition) that puts the
trivial partition at distance 1 from anything carrying flow structure — the
zero-at-trivial convention of chance-corrected label measures, without a random
model. Its base point is network-specific, so adjusted values compare partitions
of one network, not partitions across networks.
Evaluation is blockwise over the meet cells of the two partitions,
O(cells^2 + n + m^2), so it never materialises the n x n matrices. Either
partition may be hierarchical ({node: path tuple}), directed or weighted, and
nodes carrying no flow carry no weight: two partitions that differ only in where
they put isolated nodes are flow-equivalent, at D = 0 exactly.
Path data and memory networks
Edge lists forget where flow came from; path data does not have to. The
paths source curates datasets whose unit of observation is a whole
trajectory, and converts them into second-order state networks: a sliding
window turns every observed trigram i → j → k into a link from state node
(i, j) to state node (j, k), where (i, j) means "at physical node j,
having arrived from i". Infomap reads this representation natively and
finds the overlapping, memory-aware modules described in Rosvall et al.,
Memory in network flows and its effects on spreading dynamics and community
detection, Nat. Commun. 5, 4630 (2014).
from netgamut import paths
data = paths.load("wikispeedia") # 9.5 MB, 51,318 finished navigation trails
memory = data.to_state_network(order=2)
print(memory)
# StateNetwork(order=2): 4,169 physical nodes, 58,944 state nodes,
# 154,895 links from 50,509 paths
memory.to_pajek("out/") # out/wikispeedia.net: *Vertices / *States / *Links
im = memory.to_infomap(silent=True) # or straight into Infomap, directed flow
result = im.run()
The order parameter generalises the window: order=1 recovers the plain
directed network from the same trajectories, so first-order versus
second-order comparisons — the codelength case for memory — are a
one-argument change. StateNetwork.from_paths(...) accepts your own
sequences (optionally (nodes, weight) pairs) with the same machinery.
For demonstrations and small benchmarks, subset the trajectories first:
data.to_state_network(order=2, max_paths=10_000, max_nodes=300) keeps the
first 10,000 paths and the 300 most-visited nodes. Trajectories are cut at
removed nodes, never bridged across them, so no transition is fabricated.
Flight itineraries work the same way, at scale:
data = paths.load("db1b-coupon", year=2019, quarter=1) # ~202 MB zip from BTS
for airports, passengers in data.paths(): # ('OAK', 'SLC', 'LAX', ...)
...
memory = data.to_state_network(order=2)
DB1B is the US airline itinerary survey (a 10% ticket sample, public domain);
each itinerary is a sequence of airports weighted by its sampled passengers.
The raw quarterly CSV scatters an itinerary's coupons across ~2 GB, so the
first paths() call groups them once through SQLite and caches a derived
path file next to the download; later calls stream it directly. From the
shell:
netgamut get db1b-coupon --year 2023 --quarter 3
The same itineraries exist at the level of metropolitan areas, and airports have coordinates for plotting:
flights = paths.load("db1b-coupon")
where = flights.coordinates() # {'LAX': (33.94, -118.41), ...} from OpenFlights
cities = paths.load("db1b-city") # same archive, shared cache, no second download
for markets, passengers in cities.paths(): # ('31703', '30977', ...) BTS CityMarketIDs
...
cities.names()["31703"] # 'New York City, NY (Metropolitan Area)'
db1b-city names each node by its BTS CityMarketID, so JFK, LGA and EWR
are one node and an airport change inside a city collapses to one visit; the
labels stay ids (stable, and joinable with BTS's own tables) and names()
fetches the L_CITY_MARKET_ID lookup for the descriptions. Both side tables
are small and land in the cache next to the download on first use.
Two parsing choices to be aware of, both deliberate: Wikispeedia back clicks
(<, in ~17% of trails) are resolved against the browsing history, so paths
record the articles actually visited; and DB1B itineraries keep their full
airport sequence including the return legs, since that is the memory the
model is for.
Note that wikispeedia now exists in two sources — the navigation paths
here, the article hyperlink graph in SNAP — so the top-level
netgamut.load("wikispeedia") asks you to pass source="paths" or
source="snap" rather than guessing.
Higher-order data: hypergraphs and complexes
An edge list says two things met. A hypergraph says how many met at once, and
that is not recoverable afterwards — three authors on one paper and three
authors on three pairwise papers project onto the same triangle. The
ahorn source indexes the Aachen Higher-Order
Repository of Networks: 83 datasets whose unit of observation is a set of
nodes, so load returns a Hypergraph, not a Network.
from netgamut import ahorn
ahorn.search(network_type="simplicial-complex", max_nodes=1000)
ahorn.search(tag="benson") # everything from the Benson collection
ahorn.search(temporal=True) # 16 with timestamped interactions
h = ahorn.load("algebra-questions")
print(h)
# algebra-questions revision 2: 423 nodes, 1,268 interactions
for members, attributes in h.interactions(cast=int):
... # ((2, 12, 20), {'category': 'lie-algebras'})
Flattening is a choice you make, never something that happens behind your back:
h.to_networkx(projection="clique") # every pair inside an interaction; lossy
h.to_networkx(projection="star") # interactions as nodes; lossless
h.to_network() # a plain Network over the clique projection
to_infomap() uses the bipartite representation, which Infomap reads natively,
so the interaction sizes are still in the model when the modules are found —
the clique projection would have thrown them away first:
im = h.to_infomap(silent=True)
result = im.run() # modules span both nodes and interactions
Check max_interaction_size before projecting a wide one: an interaction of
k members becomes k(k-1)/2 edges, and algebra-questions has one of 107.
Revisions are pinnable, and only here
AHORN archives each revision as its own Zenodo record, so a pinned call keeps returning the same bytes after a correction lands:
ahorn.load("algebra-questions", revision=1) # reproducible
ahorn.load("algebra-questions") # latest, and not
That matters more than it sounds: revision 2 of that dataset renamed the
attribute revision 1 called genre. describe() prints the changelog, and
revisions cache side by side so pinning one does not evict another.
Node and interaction attributes
Node attributes arrive as the same NodeTable every other source produces, and
interaction attributes — which are the interesting reference labels here — come
off the interactions:
ahorn.load("senate-committees").nodes().partition("party")
# {'1': 0, '2': 0, ...} Democrat / Republican
h.interaction_labels("category") # one label per interaction, in file order
The same caveat as everywhere: an attribute is what someone recorded, not a claim about flow.
Multi-network files
Two datasets hold many networks in one file — MANTRA-2-manifolds is 43,214
triangulated manifolds, each with its own Betti numbers and orientability:
m = ahorn.load("MANTRA-2-manifolds")
m.networks()[0]
# {'id': 'manifold_2_4_5_1', 'betti_numbers': [1, 0, 1], 'name': 'S^2', ...}
m.interactions(network="manifold_2_4_5_1")
Three versions of AHORN's exchange format are in circulation — 0.1, 0.2 and the
0.3 the specification documents — and they disagree about whether the reserved
metadata keys are underscore-prefixed. The parser reads all three;
h.format_version says which one a file used, and h.file_revision is None
for the 0.1 files, which recorded no revision at all.
Edge cases worth knowing
Datasets SNAP has withdrawn
Five entries are still listed on the index page but their downloads have been
disabled (musae-facebook, twitter7, web-BeerAdvocate, web-RateBeer,
web-CellarTracker). Their metadata is kept so the catalog mirrors the site,
and load fails with an explicit message rather than a confusing 404:
snap.load("web-BeerAdvocate")
# LookupError: 'web-BeerAdvocate' has no downloadable files. SNAP has
# withdrawn a handful of datasets while keeping them listed; see ...
[d.name for d in snap.datasets() if not d.files] # check up front
Guarding against the very large ones
SNAP includes datasets in the tens of gigabytes. max_bytes refuses anything
bigger before the transfer starts:
net = snap.load("com-Friendster", max_bytes=2_000_000_000)
# DownloadError: ... is 30.1 GB, which exceeds the 1.9 GB limit.
The cache
snap.cache_dir() # ~/Library/Caches/netgamut on macOS
# ~/.cache/netgamut on Linux
snap.is_cached("ca-GrQc")
snap.clear_cache("ca-GrQc")
info = snap.cache_info()
info.bytes, info.datasets["ca-GrQc"].files
Point it somewhere else with the NETGAMUT_CACHE_DIR environment variable, or
snap.set_cache_dir(...) at runtime — useful for a shared cache on a cluster.
Downloads are published with an atomic rename and coordinated by a lock file, so concurrent processes asking for the same dataset cannot corrupt it or each other. An interrupted download leaves no partial file behind.
lfr/ sits beside the dataset sources and holds the LFR generators — the
fetched C++ and the binaries built from it. Deleting it costs one download and
a few seconds of compilation.
Keeping the catalog current
The bundled catalog is a snapshot. To pick up datasets SNAP has published since this release:
snap.refresh() # re-scrapes, stores in the cache, takes precedence
A refreshed catalog in the cache always wins over the bundled one; if it is ever unreadable the package silently falls back to the snapshot rather than breaking.
Command line
netgamut categories
netgamut list --category "Collaboration networks"
netgamut list --directed --max-edges 100000
netgamut info ca-GrQc
netgamut get ca-GrQc # prints the cached path
netgamut get algebra-questions --revision 1 # pin an AHORN revision
netgamut cache
netgamut refresh
Development
uv venv && uv pip install -e ".[all]" --group dev
pytest # offline tests only
pytest -m network # also hits snap.stanford.edu
python scripts/regenerate_catalog.py
python scripts/regenerate_ahorn_catalog.py
Each scripts/regenerate_*_catalog.py rebuilds one vendored snapshot and
refuses to write an implausible result, so an upstream redesign fails loudly
instead of shipping an empty catalog. refresh-catalogs.yml runs the SNAP,
Netzschleuder and AHORN ones monthly and opens a pull request when a collection
changes; the OGB and PyG scripts are run by hand.
The AHORN one reads YAML frontmatter out of the repository behind the site, so
it needs pyyaml — which is in the dev group and not a dependency of the
package, because only rebuilding needs it. ahorn.refresh() says so if it is
missing.
Releasing
Pushing a v* tag publishes that version to PyPI. publish.yml checks that
the tag matches __version__, runs the whole CI workflow, and uploads the
wheel and sdist that CI's build job made.
- On a branch, set
__version__insrc/netgamut/__init__.pyand rename the changelog's## Unreleasedsection to that version. Merge it to main. - Tag the merge commit on main and push the tag:
git tag -a v0.4.0 -m "0.4.0: <one-line summary>; see CHANGELOG.md"
git push origin v0.4.0
PyPI never takes the same version twice, so a bad release is fixed with a new
version, not a re-upload. If the workflow fails before the upload step, delete
the tag (git push origin :v0.4.0), fix main, and tag again.
There is no token to keep: PyPI trusts publish.yml in this repository when it
runs in the pypi environment, as set up under the project's Publishing
settings on pypi.org.
Attribution
The data belongs to the Stanford Network Analysis Project. This package only
indexes and fetches it. Please cite the datasets you use — Dataset.citations
carries the reference SNAP asks for — and cite the collection itself:
Jure Leskovec and Andrej Krevl. SNAP Datasets: Stanford Large Network Dataset Collection. https://snap.stanford.edu/data, 2014.
KONECT asks you to cite the collection alongside each network's own reference
(both are in Dataset.citations):
Jérôme Kunegis. KONECT – The Koblenz Network Collection. Proc. Int. Conf. on World Wide Web Companion, 1343–1350, 2013.
OGB and PyG are other people's collections too. Cite them if you use them:
Weihua Hu et al. Open Graph Benchmark: Datasets for Machine Learning on Graphs. NeurIPS 2020. — and the paper behind each individual dataset.
Matthias Fey and Jan Eric Lenssen. Fast Graph Representation Learning with PyTorch Geometric. ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019. AHORN is a repository in its own right, and it asks you to cite two things separately: the original source publication for a dataset — the BibTeX in
Dataset.citations, plus the Zenodo record when the exact revision matters — and the repository itself, inahorn.CITATION:
Florian Frantzen and Michael T. Schaub. A repository for discovery and reuse of higher-order network datasets. arXiv:2608.03491 (2026).
Licences on AHORN records are the upstream source's, and grant nothing beyond what that source already allowed.
generate.lfr runs Andrea Lancichinetti and Santo Fortunato's own programs,
downloaded on demand from
their MIT-licensed repository.
If you publish LFR results, cite them:
Andrea Lancichinetti, Santo Fortunato and Filippo Radicchi. Benchmark graphs for testing community detection algorithms. Phys. Rev. E 78, 046110 (2008).
Andrea Lancichinetti and Santo Fortunato. Benchmarks for testing community detection algorithms on directed and weighted graphs with overlapping communities. Phys. Rev. E 80, 016118 (2009).
License
MIT. Applies to this package, not to the datasets it downloads.
Metadata
Release files for netgamut 0.4.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| netgamut-0.4.0.tar.gz | 1.4 MB | With provenance | Details for netgamut-0.4.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| netgamut-0.4.0-py3-none-any.whl | Python 3 | none | any | Details for netgamut-0.4.0-py3-none-any.whl |
Total release size: 2.7 MB
Release files / netgamut-0.4.0.tar.gz
| Download URL | netgamut-0.4.0.tar.gz |
|---|---|
| Size | 1.4 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
730b6fa2301efe26a1c1a32f0c36aa9f37404e3572436887172291aad4642429
|
|
BLAKE2b-256 checksum How to use checksums |
418a03f84ea5c034c0dc6975f05b967767d9ae6b138a4092e21ebb108ffd8d19
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.
Transparency logRelease files / netgamut-0.4.0-py3-none-any.whl
| Download URL | netgamut-0.4.0-py3-none-any.whl |
|---|---|
| Size | 1.3 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f5a9057d68deb70151b32394003ac16023b6fb88f27597a645f8c52d7230db81
|
|
BLAKE2b-256 checksum How to use checksums |
7b45f2fd34e86cc5b2be30d5f92394bb7ad990d5d9482006701f5188c7aa7b55
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.
Transparency logSecurity for netgamut 0.4.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for netgamut-0.4.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.