Skip to main content

Accelerating Natural Gradient Descent for PINNs with Randomized Numerical Linear Algebra

arXiv

Code for the paper Accelerating Natural Gradient Descent for PINNs with Randomized Numerical Linear Algebra by Ivan Bioli, Carlo Marcati and Giancarlo Sangalli.

The repository contains

  • randngd (src/randngd): natural gradient descent (NGD) for neural PDE solvers, as preconditioned gradient descent on top of Optimistix. The linear systems with the Gramian are solved exactly (Cholesky) or with conjugate gradients, optionally preconditioned with a randomized Nyström approximation or a randomly pivoted (block) partial Cholesky factorisation (RPCholesky).
  • examples/: the PDE problems of the paper (PINNs for Poisson 3D, heat 3+1D and Kovasznay flow; FEINNs with a Deep Ritz and a dual-norm residual loss), with every method of the paper and command-line overrides of every setting.

The quasi-Newton baselines (BFGS, L-BFGS, SSBroyden with a Zoom line search) come from the companion package ssbroyden-optimistix (arXiv:2603.10599; version 0.1.0, used here: doi:10.5281/zenodo.23297645).

Installation

Python ≥ 3.11 and a JAX installation for your hardware (JAX installation guide); the paper's experiments ran in double precision on an NVIDIA RTX 4090.

pip install "jax[cuda13]"     # or the JAX variant you need
pip install randngd           # the package only

The examples are not part of the package; to run them, clone the repository:

git clone https://github.com/IvanBioli/preconditioned-natural-gradient.git
cd preconditioned-natural-gradient
pip install -e ".[examples]"  # randngd, flax, tensorboardX, scipy, ssbroyden-optimistix

Optional, for the FEINN examples:

  • the FE data are exported once with DOLFINx 0.9 (python examples/fe_assets/build_assets.py --problem poisson_2pi, and --problem canuto1_smooth), see examples/fe_assets;
  • the dual-norm residual example applies K^{-1} with NVIDIA cuDSS through spineax (the paper's runs used commit 20f2e692): pip install "git+https://github.com/Naruki-Ichihara/spineax.git@20f2e692bf8eededc78ecd893c829b79b8f41e62". --kinv dense_cholesky and --kinv host_splu work without it (with different rounding).

Quick start

python examples/poisson3d.py --method nystrom_gaussian --seed 0
python examples/poisson3d.py --method rpchol --set gamma=100 ell_max=300
python examples/kovasznay2d.py --method ngd_matfree --maxiter 200

Methods: ngd_full, ngd_matfree, rpchol, nystrom_gaussian, nystrom_sampling (NGD), bfgs, ssbroyden, lbfgs, adam (baselines). Each script's DEFAULTS block holds the settings of the paper's runs for that problem; --set FIELD=VALUE overrides any field of examples/_common/methods.py:Settings. Runs are logged to TensorBoard under --results-root (default results/), with a run_info.json of the resolved settings.

Using randngd

A metric is a function params -> Gramian; the preconditioners take it and plug into Optimistix' minimisers. Importing randngd switches JAX to double precision, which the methods need (set RANDNGD_ENABLE_X64=0 before the import to opt out); create the parameters after the import, or in float64.

import optimistix as optx
import randngd  # enables float64

# Residuals at one point, residual(params, x), and the points (N, d) and (M, d) of
# the loss  mean_i r_int(params, x_i)^2 / 2 + mean_j r_bdry(params, y_j)^2 / 2.
metric = randngd.metrics.gauss_newton(
    [(r_int, x_interior), (r_bdry, x_boundary)], params
)  # G = J_int^T J_int / N + J_bdry^T J_bdry / M, never formed

precond = randngd.NystromNGD(metric, kind="gaussian", ell_max=500, key=0)
solver = randngd.ngd_solver(precond)  # Armijo backtracking line search
sol = optx.minimise(lambda p, args: loss(p), solver, params, max_steps=1000)

randngd.metrics.gauss_newton also assembles G (representation="full", for NGDFull) or recomputes the Jacobians batch by batch ("matfree"); randngd.metrics.finite_element gives the metric J^T M J of FEINNs. Lower-level building blocks are in randngd.Gramian and randngd.jacobians.

Paper Code
Alg. 1 (NGD as preconditioned GD) randngd.optx.PreconditionedGradientDescent, randngd.ngd_solver
Alg. 2 (direct solve) randngd.NGDFull
Alg. 3 (matrix-free, CG) randngd.NGDMatrixFree
Alg. 4 (Nyström-preconditioned CG) randngd.NystromNGD
Alg. 5 (RPCholesky-preconditioned CG) randngd.RPCholNGD
Alg. 6 (Nyström sketch) randngd.nystrom
Alg. 7–8 (block RPCholesky) randngd.rpcholesky
Gramian representations (Table 1) randngd.Gramian, randngd.metrics, randngd.jacobians
Choice of the damping μ (§4.3) randngd.damping

Reproducibility

The examples run the paper's configurations by default, including a few behaviours of the original code that the libraries do not use by default (e.g. the step size in the Levenberg–Marquardt gain ratio, a minimum step size in the Armijo search); each is a named argument set in examples/_common/methods.py.

Acknowledgements

  • The collocation-point samplers and differential operators in examples/_common/unified_pinns.py are taken unchanged from the unified-pinns code of Marius Zeinhofer et al. (MIT License, LICENSE-unified-pinns).
  • randngd.optx contains code adapted from Optimistix (Apache-2.0).

License

Apache-2.0, see LICENSE.

Citation

If you use this code, please cite the paper:

@misc{bioli2025accelerating,
  title         = {Accelerating Natural Gradient Descent for {PINNs} with Randomized Numerical Linear Algebra},
  author        = {Bioli, Ivan and Marcati, Carlo and Sangalli, Giancarlo},
  year          = {2025},
  eprint        = {2505.11638},
  archivePrefix = {arXiv},
  primaryClass  = {math.NA},
  doi           = {10.48550/arXiv.2505.11638},
  url           = {https://arxiv.org/abs/2505.11638}
}

We prefer citations of the paper to citations of the code. The code is also archived on Zenodo, to refer to an exact version of it: version 0.1.0 is doi:10.5281/zenodo.23297667; all versions: doi:10.5281/zenodo.23297666.

Metadata

Release files for randngd 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for randngd 0.1.1
File Size Uploaded Provenance Actions
randngd-0.1.1.tar.gz 32.0 kB With provenance Details for randngd-0.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for randngd 0.1.1
File Interpreter ABI Platform Actions
randngd-0.1.1-py3-none-any.whl Python 3 none any Details for randngd-0.1.1-py3-none-any.whl

Total release size: 71.0 kB

Release files / randngd-0.1.1.tar.gz

Download URL randngd-0.1.1.tar.gz
Size 32.0 kB
Tags Source
SHA-256 checksum
How to use checksums
78b339ec6c10d081a94d95f6624d6ba962f30e9c90cf6092499ef40067a2047f
BLAKE2b-256 checksum
How to use checksums
1c20c1805bc18beb486f7bc9aa9194ea36b1903518644c51128157b7425730c9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 11, 2026.

Transparency log

Release files / randngd-0.1.1-py3-none-any.whl

Download URL randngd-0.1.1-py3-none-any.whl
Size 39.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
8c505792e712606f75b328722d85b008bcdb18034b72e70ce2a733d164f3db1a
BLAKE2b-256 checksum
How to use checksums
2dd31278326313072b322cd7bb5bad8771c894cb89aa70007e0eb65139e9063a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 11, 2026.

Transparency log

Security for randngd 0.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for randngd-0.1.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 11, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.1 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page