Accelerating Natural Gradient Descent for PINNs with Randomized Numerical Linear Algebra
Code for the paper Accelerating Natural Gradient Descent for PINNs with Randomized Numerical Linear Algebra by Ivan Bioli, Carlo Marcati and Giancarlo Sangalli.
The repository contains
randngd(src/randngd): natural gradient descent (NGD) for neural PDE solvers, as preconditioned gradient descent on top of Optimistix. The linear systems with the Gramian are solved exactly (Cholesky) or with conjugate gradients, optionally preconditioned with a randomized Nyström approximation or a randomly pivoted (block) partial Cholesky factorisation (RPCholesky).examples/: the PDE problems of the paper (PINNs for Poisson 3D, heat 3+1D and Kovasznay flow; FEINNs with a Deep Ritz and a dual-norm residual loss), with every method of the paper and command-line overrides of every setting.
The quasi-Newton baselines (BFGS, L-BFGS, SSBroyden with a Zoom line search) come from the companion package ssbroyden-optimistix (arXiv:2603.10599; version 0.1.0, used here: doi:10.5281/zenodo.23297645).
Installation
Python ≥ 3.11 and a JAX installation for your hardware (JAX installation guide); the paper's experiments ran in double precision on an NVIDIA RTX 4090.
pip install "jax[cuda13]" # or the JAX variant you need
pip install randngd # the package only
The examples are not part of the package; to run them, clone the repository:
git clone https://github.com/IvanBioli/preconditioned-natural-gradient.git
cd preconditioned-natural-gradient
pip install -e ".[examples]" # randngd, flax, tensorboardX, scipy, ssbroyden-optimistix
Optional, for the FEINN examples:
- the FE data are exported once with DOLFINx 0.9
(
python examples/fe_assets/build_assets.py --problem poisson_2pi, and--problem canuto1_smooth), see examples/fe_assets; - the dual-norm residual example applies
K^{-1}with NVIDIA cuDSS through spineax (the paper's runs used commit20f2e692):pip install "git+https://github.com/Naruki-Ichihara/spineax.git@20f2e692bf8eededc78ecd893c829b79b8f41e62".--kinv dense_choleskyand--kinv host_spluwork without it (with different rounding).
Quick start
python examples/poisson3d.py --method nystrom_gaussian --seed 0
python examples/poisson3d.py --method rpchol --set gamma=100 ell_max=300
python examples/kovasznay2d.py --method ngd_matfree --maxiter 200
Methods: ngd_full, ngd_matfree, rpchol, nystrom_gaussian,
nystrom_sampling (NGD), bfgs, ssbroyden, lbfgs, adam (baselines). Each
script's DEFAULTS block holds the settings of the paper's runs for that problem;
--set FIELD=VALUE overrides any field of examples/_common/methods.py:Settings.
Runs are logged to TensorBoard under --results-root (default results/), with a
run_info.json of the resolved settings.
Using randngd
A metric is a function params -> Gramian; the preconditioners take it and plug into
Optimistix' minimisers. Importing randngd switches JAX to double precision, which the
methods need (set RANDNGD_ENABLE_X64=0 before the import to opt out); create the
parameters after the import, or in float64.
import optimistix as optx
import randngd # enables float64
# Residuals at one point, residual(params, x), and the points (N, d) and (M, d) of
# the loss mean_i r_int(params, x_i)^2 / 2 + mean_j r_bdry(params, y_j)^2 / 2.
metric = randngd.metrics.gauss_newton(
[(r_int, x_interior), (r_bdry, x_boundary)], params
) # G = J_int^T J_int / N + J_bdry^T J_bdry / M, never formed
precond = randngd.NystromNGD(metric, kind="gaussian", ell_max=500, key=0)
solver = randngd.ngd_solver(precond) # Armijo backtracking line search
sol = optx.minimise(lambda p, args: loss(p), solver, params, max_steps=1000)
randngd.metrics.gauss_newton also assembles G (representation="full", for
NGDFull) or recomputes the Jacobians batch by batch ("matfree");
randngd.metrics.finite_element gives the metric J^T M J of FEINNs. Lower-level
building blocks are in randngd.Gramian and randngd.jacobians.
| Paper | Code |
|---|---|
| Alg. 1 (NGD as preconditioned GD) | randngd.optx.PreconditionedGradientDescent, randngd.ngd_solver |
| Alg. 2 (direct solve) | randngd.NGDFull |
| Alg. 3 (matrix-free, CG) | randngd.NGDMatrixFree |
| Alg. 4 (Nyström-preconditioned CG) | randngd.NystromNGD |
| Alg. 5 (RPCholesky-preconditioned CG) | randngd.RPCholNGD |
| Alg. 6 (Nyström sketch) | randngd.nystrom |
| Alg. 7–8 (block RPCholesky) | randngd.rpcholesky |
| Gramian representations (Table 1) | randngd.Gramian, randngd.metrics, randngd.jacobians |
| Choice of the damping μ (§4.3) | randngd.damping |
Reproducibility
The examples run the paper's configurations by default, including a few behaviours of
the original code that the libraries do not use by default (e.g. the step size in the
Levenberg–Marquardt gain ratio, a minimum step size in the Armijo search); each is a
named argument set in examples/_common/methods.py.
Acknowledgements
- The collocation-point samplers and differential operators in
examples/_common/unified_pinns.pyare taken unchanged from the unified-pinns code of Marius Zeinhofer et al. (MIT License, LICENSE-unified-pinns). randngd.optxcontains code adapted from Optimistix (Apache-2.0).
License
Apache-2.0, see LICENSE.
Citation
If you use this code, please cite the paper:
@misc{bioli2025accelerating,
title = {Accelerating Natural Gradient Descent for {PINNs} with Randomized Numerical Linear Algebra},
author = {Bioli, Ivan and Marcati, Carlo and Sangalli, Giancarlo},
year = {2025},
eprint = {2505.11638},
archivePrefix = {arXiv},
primaryClass = {math.NA},
doi = {10.48550/arXiv.2505.11638},
url = {https://arxiv.org/abs/2505.11638}
}
We prefer citations of the paper to citations of the code. The code is also archived on Zenodo, to refer to an exact version of it: version 0.1.0 is doi:10.5281/zenodo.23297667; all versions: doi:10.5281/zenodo.23297666.
Metadata
Release files for randngd 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| randngd-0.1.1.tar.gz | 32.0 kB | With provenance | Details for randngd-0.1.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| randngd-0.1.1-py3-none-any.whl | Python 3 | none | any | Details for randngd-0.1.1-py3-none-any.whl |
Total release size: 71.0 kB
Release files / randngd-0.1.1.tar.gz
| Download URL | randngd-0.1.1.tar.gz |
|---|---|
| Size | 32.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
78b339ec6c10d081a94d95f6624d6ba962f30e9c90cf6092499ef40067a2047f
|
|
BLAKE2b-256 checksum How to use checksums |
1c20c1805bc18beb486f7bc9aa9194ea36b1903518644c51128157b7425730c9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 11, 2026.
Transparency logRelease files / randngd-0.1.1-py3-none-any.whl
| Download URL | randngd-0.1.1-py3-none-any.whl |
|---|---|
| Size | 39.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
8c505792e712606f75b328722d85b008bcdb18034b72e70ce2a733d164f3db1a
|
|
BLAKE2b-256 checksum How to use checksums |
2dd31278326313072b322cd7bb5bad8771c894cb89aa70007e0eb65139e9063a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 11, 2026.
Transparency logSecurity for randngd 0.1.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for randngd-0.1.1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 11, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.