pinnmark 0.1.0

PINNMark inverse PDE benchmarking toolkit and CLI facade

PINNMark

pinn_bench is a governed, reviewable benchmark system for PINN and operator-ready AI4S evaluation, anchored on a frozen formal baseline, fair-budget nominal benchmarking, structured OOD benchmarking, comparison-rights methodology, and bounded benchmark recoverability.

PyPI package:

pip install pinnmark

Facade usage:

from pinnmark import run_full_inverse_benchmark

run_full_inverse_benchmark.main(["--help"])

Project layers:

• pinn_bench core is the governed benchmark OS
• pinn_bench.easy is the sklearn-like researcher front door
• PINNAtlas is the evidence/publication product built on top of that OS

Start Here

• I need benchmark truth / release semantics. Layer 1. Goal: understand the benchmark OS, the current canonical roots, and the governed paper-facing benchmark identity. First destination: docs/product/getting_started.md Second destination: docs/product/core_story.md, docs/product/project_positioning.md
• I need Python-first research workflow. Layer 2. Goal: run, inspect, compare, study, or sweep from the researcher front door without entering governance surfaces first. First destination: docs/easy/index.md Second destination: docs/easy/quickstart.md, docs/easy/choose_guide.md
• I need review / paper-facing benchmark package. Layer 1. Goal: inspect the current NeurIPS-facing benchmark-system package, evidence path, and reviewer route. First destination: docs/product/reviewer_summary.md Second destination: paper/neurips2026/paper_reviewer_safe_summary.md, docs/product/paper_submission_package.md
• I need PINNAtlas corpus / method atlas / supplement / TPAMI package. Layer 3. Goal: enter the companion evidence/publication product without confusing it with the runtime system. First destination: docs/product/pinnatlas_program.md Second destination: paper/pinnatlas_supplement/README.md, paper/pinnatlas_tpami/README.md

Easy Path (Python-first researcher core)

The Python-first Easy Path provides one unified researcher core: the six easy objects ([ModelSpec, TaskSpec, BenchmarkEstimator, BenchmarkSuite, BenchmarkReport, BenchmarkComparison]) behave like a sklearn estimator graph with shared config discovery, fitting, and inspection semantics. Every BenchmarkEstimator still compiles down to ExperimentConfig, runs through the standard runner, and writes the governance artifacts (config_snapshot.yaml, metrics.json, train_log.csv, model.pt, loss_curve.png). The easy layer is additive: it exposes a single researcher-front door while leaving CLI commands, sectioned configs, and governance artifacts untouched.

from pinn_bench.easy import BenchmarkEstimator

est = BenchmarkEstimator(
    task="heat1d",
    model="mlp",
    epochs=5,
    lr=1e-3,
    metrics=("mse", "relative_l2"),
)
est.fit(output_dir="outputs_playground/easy_nominal")
print("Score:", est.score())
print("Report:", est.report().summary())

Need to tweak robustness, comparison, or metrics? The Easy Path helpers clone the estimator to adjust parameters, so you keep a chainable API that still resolves to a valid config:

est.with_robustness(protocol="noise", level=0.05).fit()
est.with_comparison(
    baseline_family="pinn",
    data_budget_class="sparse",
    compute_budget_class="medium",
    tuning_budget_class="fixed_recipe",
    comparison_scope="matched_all",
).score()

For small multi-run research workflows, stay in the same object graph with BenchmarkSuite and BenchmarkComparison instead of dropping into a separate toolchain immediately, and use the starter catalog (all_starters() / describe_starter(...)) when you need ready-made task+model+profile combos.

See docs/easy/quickstart.md, docs/easy/python_api.md, docs/easy/which_path_should_i_use.md, docs/easy/compatibility_policy.md, docs/easy/error_guide.md, and docs/easy/sklearn_gap_scorecard.md for the easy-path workflow guidance, and examples/easy for runnable recipes (single run, starter run, comparison, study, sweep, inspection). Use describe_model(...) and describe_task(...) when you want the docs card, starter configs, source path, and parameter signature for a component before running it, and use describe_metric(...) when you only need the metric source path and callable signature.

External Model Integration

PINNMark now keeps a unified native + external model universe with:

• a checked-in external family registry
• adapter-backed first-wave PINN family integration
• source benchmark / state / capability / provenance metadata

Canonical-first discovery examples:

pinn-bench list families --json
pinn-bench list families --name pinnmark.pinn.vanilla --json
pinn-bench list models --external-only --source-benchmark PINNacle --json
pinn-bench list models --name PINNacle::fnn --json

Python equivalents:

from pinn_bench.easy import all_external_models, all_families, describe_family, describe_model

print(all_families())
print(describe_family("pinnmark.pinn.vanilla"))
print(all_external_models(source_benchmark="PINNacle"))
print(describe_model("PINNacle::fnn"))  # lineage / variant lookup

Current integration boundary:

• some external families are runnable through thin adapters or native proxies
• some are registry-only or adapter-ready but intentionally not runnable yet
• the registry is broader than the runnable subset by design

Canonical Family Layer

PINNMark now distinguishes three layers:

1. canonical family
   • benchmark-facing identity
   • example: pinnmark.pinn.vanilla
2. implementation variant
   • runnable or reference implementation backend
   • example: native.mlp, external.pdebench.pinn_proxy
3. provenance / lineage
   • source benchmark or source family reference
   • example: PDEBench::pinn, PDENNEval::PINN, PINNacle::fnn

This is not just aliasing. It means the platform can now say:

• these original source families collapse into one canonical PINNMark family
• but their provenance and fidelity differences are still preserved below that identity

See:

• docs/models/native_family_internalization.md
• artifacts/benchmark_model_inventory/canonical_family_collapse_map.csv
• artifacts/benchmark_model_inventory/native_family_internalization_status.csv
• paper/neurips2026/canonical_family_wording_migration_report.md
• paper/neurips2026/canonical_family_authoring_style_guide.md

Canonical-family discovery examples:

pinn-bench list families --json
pinn-bench list families --name pinnmark.pinn.vanilla --json

from pinn_bench.easy import all_families, describe_family

print(all_families())
print(describe_family("pinnmark.operator.fno"))

Canonical-family wording rule:

• README, docs, paper summaries, figure titles, and table method columns should lead with canonical family ids such as pinnmark.pinn.vanilla and pinnmark.operator.fno.
• External names such as PDEBench::pinn, PDENNEval::PINN, and PINNacle::fnn are still kept, but as lineage / provenance notes rather than the primary benchmark-facing method names.
• If a paper or appendix surface needs both layers, use the canonical family in the main label and move the external name into a lineage note, provenance clause, or dedicated mapping column.

Primary Method Layer

PINNMark now also tracks a benchmark-primary method universe above raw runtime selectors.

• top-level families remain the stable long-horizon taxonomy
• primary methods are the author-facing benchmark method objects
• runtime selectors remain the concrete constructible defaults when they exist

Examples:

• pinnmark.pinn.cpinn
• pinnmark.hybrid.fbpinn
• pinnmark.pinn.vpinn
• pinnmark.pinn.hpinn
• pinnmark.operator.fno
• pinnmark.operator.deeponet
• pinnmark.operator.pino
• pinnmark.multifidelity.mf_pidnn

Primary-method discovery examples:

pinn-bench list methods
pinn-bench list methods --name cPINN --json

from pinn_bench.easy import all_primary_methods, describe_primary_method

print(all_primary_methods())
print(describe_primary_method("cPINN"))

Primary-method implementation report:

• paper/neurips2026/twenty_method_internalization_report.md

Suite Path (NeurIPS 2026 first wave)

Need one suite-level front door instead of piecing together matrix / transfer / reference runners by hand?

pinn-bench suite list
pinn-bench suite run --suite inverse_main_benchmark --family-id inverse_main_benchmark__allencahn1d__parameter_recovery__epsilon_family_v1__mlp__physics_on --seeds 0
pinn-bench suite inspect --suite inverse_main_benchmark

Or stay in Python with the unified suite object:

from pinn_bench import Suite

suite = Suite.load("small_sample_inverse_suite")
result = suite.run(dry_run=True)
summary = suite.summary()
print(summary.execution_state, summary.figure_state)
print([figure.figure_id for figure in suite.figures()])
print([claim.claim_id for claim in suite.claims()])

Use paper/neurips2026/zero_to_suite_quickstart.md for the shortest “run one suite, find the results, find the figure, find the claim” path.

External Variant Lookup

PINNMark still supports external-variant lookup when authors or reviewers need to trace runnable proxies and source-benchmark lineage.

Useful discovery paths:

pinn-bench list models --json
pinn-bench list models --external-only --json
pinn-bench list models --external-only --source-benchmark PINNacle --json
pinn-bench list models --external-only --integration-state external_runnable --json
pinn-bench list models --external-only --runnable-only --json

Python discovery:

from pinn_bench.easy import all_external_models, describe_model

print(all_external_models(source_benchmark="PINNacle"))
print(describe_model("PINNacle::fnn"))

Implementation anchors:

• pinn_bench/models/external_registry.py
• pinn_bench/models/external_adapters/
• artifacts/benchmark_model_inventory/external_model_integration_status.csv

Current boundary:

• not every external family is runnable
• runnable families are adapter-backed, not full native rewrites of the upstream benchmarks
• blocked or registry-only families are still discoverable and provenance-traceable

Current implementation report:

• paper/neurips2026/external_model_integration_report.md

Current machine-readable status export:

• artifacts/benchmark_model_inventory/external_model_integration_status.csv
• artifacts/benchmark_model_inventory/external_model_integration_status.json

PINNAtlas Companion

PINNAtlas is not the runtime front door. It is the companion evidence/publication product layered on top of pinn_bench. Use it when you need field-scale corpus, audited claim, method-atlas, supplement, or TPAMI package surfaces.

A reproducible PINN benchmark toolkit with frozen formal baselines, a current six-suite execution truth layer, reviewer-facing bundle controls, and explicit formal-freeze workflows.

Core Claim Surfaces

1. governed nominal benchmarking
2. structured OOD benchmarking
3. comparison-rights methodology
4. benchmark recoverability

Supporting Mechanisms

• stable truth and governance
• DX and role-specific entry surfaces
• operator-ready abstraction without silent relabeling
• synthesis and packaging for external inspection

Not This Project

• not a Main Track algorithm paper
• not a hosted leaderboard or hosted service
• not a complete geometry benchmark
• not a complete inverse-trust benchmark
• not a complete operator benchmark
• not a participant-validated recoverability paper
• not an all-AI4S breadth paper

Fast Start

1. Inspect repo truth:

pinn-bench version --json

2. List the current suite front door:

pinn-bench suite list

3. Inspect the flagship current suite:

pinn-bench suite inspect --suite inverse_main_benchmark

4. Read the current execution truth: paper/neurips2026/current_execution_result_audit.md
5. Read the reviewer-safe package route: paper/neurips2026/paper_reviewer_safe_summary.md, paper/neurips2026/submission_bundle_manifest.md, docs/product/paper_submission_package.md

Principal 50-Seed Package

For a seed-expanded principal inverse package that keeps the current 20-seed submission route intact, materialize a separate package directory:

PYTHONPATH=. python3 scripts/run_full_inverse_benchmark.py \
  --seeds 50 \
  --stress-seeds 50 \
  --flagship-seeds 50 \
  --outdir artifacts/submission_50seed \
  --figdir artifacts/submission_50seed/figures

This package writes:

• principal per-seed metrics, main-board summaries, sidecar/plausibility metrics, and audit summaries under artifacts/submission_50seed/
• bounded OOD and flagship summaries under the same package root
• LaTeX snippets under artifacts/submission_50seed/latex/
• a package README at artifacts/submission_50seed/README.md

Reviewer/user navigation assets for the current branch live at:

• paper/neurips2026/user_playbook_quick_reference.md
• paper/neurips2026/adoption_checklist.md
• paper/neurips2026/method_family_lane_guide.md
• paper/neurips2026/scenario_matrix_coverage.md

Reader Paths

Supporting navigation:

• first-time reader: docs/product/core_story.md, docs/product/why_pinn_bench.md, docs/product/getting_started.md
• benchmark user: paper/neurips2026/zero_to_suite_quickstart.md, paper/neurips2026/current_execution_result_audit.md
• benchmark contributor: docs/product/contribution_path.md, paper/neurips2026/execution/real_result_inventory.md, paper/neurips2026/execution/suite_status_board.csv
• governance / review reader: docs/product/reviewer_summary.md, paper/neurips2026/paper_reviewer_safe_summary.md, docs/product/paper_submission_package.md, paper/neurips2026/submission_bundle_manifest.md

Highlights

• Frozen formal benchmark lines for v0.1, v0.2, v0.3, v0.4, v0.5, v0.6, and v1.0
• Unified pinn-bench CLI for runs, matrix execution, reporting, presets, demos, scaffold flows, doctor checks, summaries, audits, formalization, and community bundle workflows
• Current 2026 paper route is anchored on one frozen fair-board layer, one current six-suite executed extension line, and one reviewer-safe truth chain
• Comparison-rights support remains bounded and reviewer-legible through current paper/package surfaces rather than preview-era launch routing
• Recoverability remains bounded and not participant-validated; no completed B1 participant outcome ledger is checked into current truth
• Reviewer-facing bundle controls, metadata routing, and claim-evidence crosswalks aligned to the NeurIPS Datasets & Benchmarks paper package
• Additive v3.0-preview super-benchmark registry scaffolding for strict official/staging/evidence cohort governance across absorbed SciML sources

Install

pip install pinnmark

# local development install
pip install -e .

Minimum Python version: 3.9

Core runtime dependencies:

• numpy
• pandas
• matplotlib
• tqdm
• PyYAML
• torch
• scipy

The PyPI-facing CLI is pinnmark-run. The repository-native CLI pinn-bench remains supported, including python -m pinn_bench.run --config ....

Reproducibility note:

• benchmark outputs depend on the selected config, explicit seed, and available hardware/runtime backend
• for reproducible runs, prefer passing explicit seeds and retaining config_snapshot.yaml, metrics.json, and train_log.csv

Version State

• v0.1: frozen historical baseline for mlp + siren on outputs/
• v0.2: frozen historical baseline for mlp + siren + kan on outputs_v02/
• v0.3: frozen historical baseline for the 33-family benchmark core and first product-layer formalization, on outputs_v03/
• v0.4: frozen historical baseline for the 47-family union baseline and first community-layer formalization, on outputs_v04/
• v0.5: frozen historical baseline for the thin-delta submission/community/platform release, on outputs_v05/
• v0.6: frozen historical operator-ready formal baseline, on outputs_v06/
• v1.0: current frozen formal baseline, on outputs_v10/
• current preview line: none
• latest completed preview line: v1.9-preview
• archived preview provenance docs: docs/product/v1_9_preview.md, docs/product/v1_9_scope.md, docs/product/v1_9_governance.md, docs/product/v1_9_closeout.md
• previous completed preview docs: docs/product/v1_8_preview.md, docs/product/v1_8_scope.md, docs/product/v1_8_governance.md, docs/product/v1_8_closeout.md
• freeze completion record: docs/product/v1_0_freeze_complete.md

Formal v1.0 source-of-truth assets:

• manifest: configs/matrix/baseline_v1_0_formal.yaml
• canonical formal root: outputs_v10/

Researcher Path

Formal v1.0 workflow

Use the current frozen formal manifest and formal outputs root:

pinn-bench formalize --from-preview v0.9-preview --source-root outputs_preview_v09 --target-root outputs_v10 --target-version v1.0 --dry-run
pinn-bench formalize --from-preview v0.9-preview --source-root outputs_preview_v09 --target-root outputs_v10 --target-version v1.0
pinn-bench doctor --formal --outputs-root outputs_v10 --matrix-manifest baseline_v1_0_formal.yaml

# read-only doctor verification
pinn-bench doctor --formal --outputs-root outputs_v10 --json

# explicit doctor artifact refresh
pinn-bench doctor --formal --outputs-root outputs_v10 --json --write
pinn-bench doctor --formal --outputs-root outputs_v10 --write

# read-only audit verification
pinn-bench audit --formal v1.0 --outputs-root outputs_v10 --json

# explicit audit artifact refresh
pinn-bench audit --formal v1.0 --outputs-root outputs_v10 --json --write
pinn-bench audit --formal v1.0 --outputs-root outputs_v10 --write

Archived Preview Provenance

The older preview closeout package is still preserved for historical traceability.

Use these archived surfaces only when you need preview-era provenance rather than the current 2026 paper route:

• docs/product/v1_9_preview.md
• docs/product/v1_9_scope.md
• docs/product/v1_9_governance.md
• docs/product/v1_9_closeout.md
• docs/product/public_quickstart.md

Community Path

Package, validate, compare, and import one official run bundle against the current formal line:

pinn-bench package-run outputs_v06/heat1d_mlp_seed0 --output-dir outputs_playground/example_bundle
pinn-bench validate-submission outputs_playground/example_bundle --against-manifest baseline_v1_0_formal.yaml
pinn-bench compare-bundle outputs_playground/example_bundle --against outputs_v10/_leaderboard/leaderboard_grouped.json
pinn-bench import-bundle outputs_playground/example_bundle --against outputs_v10/_leaderboard/leaderboard_grouped.json --output-dir outputs_playground/imported_bundle_views/example_bundle

Build the additive v3.0-preview super-benchmark registry from absorbed source packs:

pinn-bench superbench-registry --source-root examples/superbench_v3_preview/sources --output-dir /tmp/pinn_superbench_registry

Run one absorbed source entry through the native rerun + bundle + registry-sync path:

pinn-bench superbench-rerun-entry   --source-record examples/superbench_v3_preview/sources/PINNacle/source_record.json   --entry-id PINNacle::poisson2d::mlp::official_rerun   --workspace-root examples/superbench_v3_preview/sources/PINNacle   --registry-root /tmp/pinn_superbench_registry

Run one ordered SciML tranche through the same orchestration path:

pinn-bench superbench-run-tranche   --tranche-manifest examples/superbench_v3_preview/tranches/sciml_platform_tranche_01.json   --registry-root /tmp/pinn_superbench_registry

Docs Hubs

• Product docs hub: docs/product/index.md
• Experiment docs hub: docs/experiments/index.md
• Community docs hub: docs/community/index.md
• Paper packaging hub: docs/product/paper_submission_package.md

Historical Formal Releases

• docs/experiments/v0_6_release.md
• docs/experiments/v0_6_runbook.md
• docs/experiments/v0_5_release.md
• docs/experiments/v0_5_runbook.md
• docs/experiments/v0_4_release.md
• docs/experiments/v0_4_runbook.md
• docs/experiments/v0_3_release.md
• docs/experiments/v0_3_runbook.md