nirs4all-repository

Status: 0.1.0 beta. The storage envelope and the cross-language index.json contract are frozen at schema_version: 1. Catalogue website: repository.nirs4all.org.

The public, versioned repository of pre-configured, tested nirs4all pipelines — ready-to-run recipes (preprocessing → model → evaluation) stored with provenance, validated against reference datasets, and loadable by name instead of rebuilt by hand. It is the remote, name-addressable layer the ecosystem otherwise lacks.

Part of the nirs4all ecosystem. See also the public open-source NIRS tools hub for the file readers, datasets, methods engine, browser modelling app, reproducible pipeline repository, papers archive, benchmarks, and release cockpit.

What it does

• Store pipelines — nirs4all recipes or dag-ml recipes, with or without fitted artifacts — as portable, content-addressed bundles with full provenance.
• Validate them statically (schema, structure, checksums) and functionally (re-run against a pinned reference dataset and compare to recorded metrics).
• Secure them — recipe class allow-listing, pickle-opcode scanning, SHA-256 tamper-evidence, trust tiers, and a publication gate.
• Serve them by name through a language-agnostic static contract (a canonical-JSON index.json + content-addressed bundles over HTTPS) consumed by the Python reference client and thin clients in any language.
• Publish a catalogue website listing every pipeline, its metadata, provenance, and validation status.

Install & use

pip install nirs4all-repository

Install the optional execution/strict-validation integration with:

pip install "nirs4all-repository[nirs4all]"

That extra supports the existing nirs4all 0.10–0.12 line, R1 (0.13.0), and the R2/R3 1.0.0rc* train. Repository itself remains lightweight and can browse or fetch recipes without installing nirs4all.

import nirs4all_repository as n4r

n4r.list(framework="nirs4all")        # browse the catalogue
pipe = n4r.get("snv_savgol_pls")      # resolve by name (local → bundled → remote, sha256-verified)
config = pipe.to_nirs4all()           # hand to nirs4all.run() / predict()

For provider/core clients that want explicit preset/pipeline names, get_pipeline_list(), get_pipeline(), and get_bundle() are the stable provider-facing entry points for the same read-only catalogue semantics.

The n4a-repository CLI is the maintenance interface (list / show / get / add / validate / scan / build / site / evaluate / publish).

How a pipeline is stored

Each pipeline is a directory pipelines/<id>/:

The generated catalog/index.json is the single cross-language registry, served at /data/index.json and bundled into the wheel for offline resolution. The envelope is dag-ml's Research-Provenance-Package shape, so one format serves both ecosystems.

See docs/DESIGN.md, docs/SPECIFICATION.md, and docs/ROADMAP.md. Security model: SECURITY.md.

Relationship to the rest of the ecosystem

• Recipes are expressed against the public nirs4all pipeline surface (or the dag-ml DSL) and executed by their engines; this repo does not re-implement NIRS, IO, or ML logic.
• They are scored and ranked in nirs4all-benchmarks; this repo is the source of the recipes, the benchmarks repo is where they are run and ranked.
• Reference datasets come from nirs4all-datasets (by name / DOI); never re-hosted here.

Development (green gate)

ruff check .
mypy src/nirs4all_repository
n4a-repository validate --all
n4a-repository build && git diff --exit-code   # generated artifacts are current
pytest -m "not network and not evaluate"

Release hygiene

GitHub Releases are expected to carry the same wheel and sdist built by the release workflow, so clients that fall back to GitHub Release assets do not land on source-only tags. On release.published, .github/workflows/publish.yml builds dist/, uploads those files to the GitHub Release, and verifies that a *.whl and *.tar.gz are attached. Manual dry-runs remain build-only.

To audit an existing release without publishing or touching PyPI:

python scripts/check_release_assets.py --repo GBeurier/nirs4all-repository
python scripts/check_release_assets.py --repo GBeurier/nirs4all-repository --tag v0.1.8

The check uses GH_TOKEN/GITHUB_TOKEN or local gh credentials when available, to avoid anonymous GitHub API rate limits.

License

Pipeline code / configurations are dual-licensed open-source — CeCILL-2.1 OR AGPL-3.0-or-later. The copyleft terms permit commercial use; an optional commercial license is available for proprietary/closed-source use that cannot meet the copyleft obligations (contact nirs4all-admin@cirad.fr). Bundled content (cards, metadata, the website) is CC-BY-4.0. See LICENSING.md and LICENSES/.