ProdNet Generator — firm-level production-network reconstruction toolkit

prodnet_generator builds a firm-level weighted production network for a whole economy (up to millions of firms) out of nothing more than published, sector-level data:

• a firm-size distribution (how many firms of each receipt-size sit in each industry), and
• a national input–output (IO) table (which sectors buy from which).

From those two aggregate objects it reconstructs who links to whom and how many dollars flow on each edge, using a gravity–Bernoulli fitness kernel (a hidden-variable link model with firm size as the hidden variable, calibrated to the IO margins), then closes the network to be irreducible + aperiodic and solves the dollar weights with a bundled C ADMM min-energy solver.

It is solver-license-free: the one-time λ-fit for a new economy runs on IPOPT behind the free MUMPS linear solver, and the dollar-weight solve is the bundled mew C program. Reconstructing an economy whose λ is already fitted (e.g. the bundled demos) needs no NLP solver at all.

Formerly netrecon; import netrecon remains a permanent compatibility alias.

---

Install

macOS on Apple silicon, Python 3.11–3.13. Install into a virtual environment (recommended — it sidesteps the system-Python externally-managed-environment error):

python3 -m venv env && source env/bin/activate && pip install prodnet-generator

That single pip install pulls one wheel that bundles the whole pipeline — the stage modules, the mew weight-solver binary, and two demo economies (Denmark + Finland) — so it reconstructs networks out of the box:

import prodnet_generator
net = prodnet_generator("Denmark", n_firms=8000, seed=1)   # -> Network (deterministic)
net.n_firms, net.n_edges                                   # (7975, 436633)

Platform: the candidate engine is pure Python (numba-JIT), so one wheel serves CPython 3.11 / 3.12 / 3.13. It carries the arm64 mew binary, so it is macOS/Apple-silicon only — pip cleanly refuses it on Linux/Intel; there, use the repository (developer) install below.

Bundled vs. on-demand economies

The wheel ships two demo economies ready to run — Denmark and Finland. The four large economies (US, Japan, United_Kingdom, Australia) are not shipped in the wheel; to reconstruct them from a pip install, point $PRODNET_DATA_DIR at a directory holding their <C>_interfirm bundles (both the library and the pipeline honor it):

export PRODNET_DATA_DIR=/path/to/economy_bundles   # contains US_interfirm/, Japan_interfirm/, ...

Developer / from-source install

A repository checkout runs its own code_generation/ pipeline (the byte-identical "golden" path) and carries all six economies under code_generation/datasets/:

git clone git@bitbucket.org:VipinVeetil/prodnet-generator.git
cd prodnet-generator
python3 -m venv .venv && source .venv/bin/activate   # Python 3.11–3.13
pip install -e .                     # editable dev install

Fitting a new economy's λ-family additionally needs IPOPT behind cyipopt (pip install 'prodnet-generator[fit]', plus brew install ipopt or conda install -c conda-forge ipopt mumps if pip can't supply one). A licensed HSL / MA97 solver is optional (faster on hard fits): export IPOPT_LINEAR_SOLVER=ma97.

---

Quickstart

The Python library (import prodnet_generator)

import prodnet_generator

prodnet_generator.available_economies()               # which economies are ready right now
prodnet_generator.capabilities()                      # what can THIS machine reconstruct?

net = prodnet_generator("Denmark", n_firms=8000, seed=1)   # the module is callable
net.n_firms, net.n_edges                                   # (7975, 436633)
W = net.to_scipy()                                         # scipy CSR: W[buyer-1, seller-1] = spend share
net.save("my_denmark_net")                                 # persist weights.csrbin + firm maps

prodnet_generator(...) forwards to prodnet_generator.generate(...); reconstruct is the same function under its pre-2.0 name, and import netrecon re-exports the whole surface unchanged.

n_firms is the primary knob (converted internally to the census subsample fraction; seed fixes everything, so the same call is byte-identical every time). The library is hardware-aware: before a large run it estimates peak RAM and, if your machine can't fit it, refuses with the largest feasible n_firms — or pass autoscale=True to auto-fit, force=True to override. The same API is callable identically from Python, Cython and C++ (see docs/40_interop.md).

The GUI

The pip install provides console scripts — launch the browser app with:

prodnet-gui        # aliases: prodnet-run, netrecon-gui, netrecon-run

It starts a local Flask server and opens your browser: pick an economy and scale, tweak the knobs, and download the resulting weights.csrbin. See docs/GUI.md.

The CLI (repository checkout)

python code_generation/run_pipeline.py --economy Denmark_interfirm --gamma 0.02 --seed 1 --bin --cpu-draw

--gamma is the subsample fraction of the census. Output lands in outputs/pipe_<tag>/weights.csrbin. Every stage and knob is documented in docs/PIPELINE.md.

---

What's in the box

Component	What it is
Reconstruction pipeline (code_generation/run_pipeline.py + stages)	Turns firm-size + IO into a weighted, irreducible, aperiodic firm network. Bundled inside the wheel under prodnet_generator/_pipeline/.
Python library (import prodnet_generator)	The pip-installable, callable API over the pipeline: prodnet_generator(economy, n_firms=…), capabilities() / feasibility() hardware checks, and two packaged demo economies. Deterministic and callable identically from Python, Cython and C++.
ProdNet Generator GUI (code_generation/prodnet_generator/)	A local Flask browser app — a friendly front end for the pipeline (prodnet-gui): presets, knobs, and a results viewer.
ESRI research engines (code_else/)	Firm-level systemic-risk cascades (Linear, Leontief, CES-granular) over a reconstructed network. These are research scripts in the repository only — they are not part of the installed library or GUI.

---

Repository layout

prodnet_generator/          the pip package (library)
  api.py                      the callable facade over the pipeline
  hardware.py                 hardware-aware preflight (capabilities/feasibility)
  _pipeline/                  the pipeline bundled into the wheel (code + demo data)

code_generation/            the reconstruction pipeline (dev / golden path)
  run_pipeline.py             Stage B→G driver (solve z → draw → floor → closure → weights)
  census_estimate.py          Stage 1 FlowFit (λ on representative firms) + subsampling
  step2_gravity_bernoulli.py  Stage 1c CPU Bernoulli draw
  gpu_stage_c.py              Stage 1c GPU (Metal/MLX) Bernoulli draw
  step3_degree_floor.py       Stage D: degree-floor repair
  step4_make_irreducible_aperiodic.py  Stages E/F: single-SCC + aperiodic closure
  markov_closure/             Stage 2 "mew" — the C ADMM min-energy dollar-weight solver
  recon_bundle.py             resolve an economy name -> its validated (census, λ, IO) trio
  read_weighted_csr.py        loader for weights.csrbin (CSR / COO)
  datasets/<C>_interfirm/     consistent-model bundle: census.npz, lambda.npz, flow_target_F.txt, input_files/
  prodnet_generator/          the Flask GUI (app.py, runner.py, templates/)

code_else/                  research layer (ESRI engines; not part of the library)

---

Documentation

Doc	Contents
Library guide — docs/00_index.md → docs/70_faq.md	The pip-installable API: quickstart, method + math, hardware-awareness, interop, cookbook, API reference, FAQ. Also a self-contained offline HTML site at docs/html/index.html.
docs/ARCHITECTURE.md	The pipeline as a data-flow: inputs → Stage 1 fit → draw → closure → Stage 2 weights.
docs/PIPELINE.md	run_pipeline.py — every stage and every CLI knob with its locked default.
docs/GUI.md	The ProdNet Generator GUI: launch, presets, knobs, estimate-α, results viewer, routes.
docs/INPUT_FORMAT.md	The input CSVs the GUI/pipeline consume — columns + examples.

---

Authors & license

Created by Ashwin Bhattathiripad and Vipin P Veetil (https://www.vipinveetil.com). Released under the MIT License — see LICENSE.