nirs4all-tools

Offline, one-way, no-in-place migration tools for legacy nirs4all artifacts (workspaces, .n4a bundles, loose prediction files).

This is the standalone home for the legacy readers that used to live inside the nirs4all runtime. The V1 runtime carries no legacy reader and no auto-migration trigger; instead, nirs4all-tools converts old stores into the format the runtime already reads (nirs4all-workspace-v2), so users keep their predictions/pipelines without the runtime ever opening a legacy store.

Status: first transform (lane L18, lock LOCK-MIG, decision DEC-MIG-001). Candidate packages are unpublished; install a wheel supplied through the release process rather than assuming a package-registry version exists. The CLI surface, the no-in-place safety machinery, detection, the contract vocabulary, inspect, migrate --dry-run, and --copy-only are implemented. The first schema transform lowers sqlite-workspace-legacy-arrays metadata into a fresh workspace-v2 store.sqlite; legacy array rows are lowered into runtime-readable arrays/<dataset>.parquet sidecars when the optional parquet extra is installed, and the raw rows are still preserved as checksummed JSONL audit provenance. A native-results-v1 preview can lower one current dag-ml native results directory into runtime-readable workspace-v2 metadata plus array sidecars after strict hash/schema preflight. One closed historical DuckDB profile can lower a validated multi-run/pipeline/chain graph with flat arrays after read-only schema and relation preflight. A legacy runs/*/*/manifest.yaml preview can lower one completed run when it references one complete *_predictions.json payload and the YAML/JSON metadata agree.

The one contract: no-in-place

Every command guarantees the source is never modified:

• the source is opened read-only (SQLite via file:…?mode=ro&immutable=1);
• --output is mandatory and must be disjoint from the input (aliasing / nesting is refused, exit 40);
• the output must be empty; --resume is only a read-only, attested no-op for an already complete output, never a continuation of a partial migration;
• inspect and migrate materialize a private descriptor-bound, no-follow source view before detection, parsing, transforms, or payload copies; the original source is rechecked before completion (a mismatch is exit 70);
• that secure materialization requires POSIX O_NOFOLLOW, O_DIRECTORY, and descriptor-relative (dir_fd) operations. Where they are unavailable, source-consuming commands fail closed with unsupported capability rather than fall back to path traversal. It needs temporary space roughly equal to the source, and TMPDIR must resolve outside both source and output.

Install

pip install -e ".[dev]"          # scaffold core is pure standard library
pip install -e ".[duckdb]"       # add DuckDB-source reading (optional)
pip install -e ".[parquet]"      # add Parquet lowering/validation (optional)
pip install -e ".[trusted-joblib]" # explicit, trusted sklearn PLS inspection only
pip install -e ".[trusted-joblib,n4mm-export]" # opt-in trusted PLS -> PREDICT-only N4MM export

CLI

For an operator procedure that needs no source checkout, including the R2 to R1 rollback path, see Workspace conversion runbook. The version-based read/write/migrate commitment and exact per-input dispositions are defined by the legacy support SLA and its machine-readable matrix.

nirs4all-tools --version

# Read-only: detect what a legacy location contains.
nirs4all-tools workspace inspect INPUT [--format json|text] [--report PATH]

# Convert into a fresh workspace-v2 output (one-way, no-in-place).
nirs4all-tools workspace convert INPUT --output OUTPUT [--dry-run | --verify]

# Historical/advanced interface, retained unchanged.
nirs4all-tools legacy inspect <input> [--format json|text] [--report PATH]
nirs4all-tools legacy migrate <input> --output DIR --target nirs4all-workspace-v2 \
    [--manifest PATH] [--report PATH] [--id-map PATH] [--unsupported-report PATH] \
    [--checksums sha256] [--dry-run | --verify] [--strict | --best-effort] \
    [--copy-only] [--resume] [--trusted-load-joblib]

# Verify an output against its manifest (reads no source).
nirs4all-tools legacy verify <output-dir> --manifest PATH [--report PATH]

# Deliberately narrow: produces model.n4mm plus an attestation, never a
# workspace or archive.  joblib is deserialized only with this explicit flag.
nirs4all-tools legacy export-n4mm <trusted-pls.joblib> --output DIR --trusted-load-joblib

workspace inspect is read-only. workspace convert fixes the target to nirs4all-workspace-v2 and returns the same stable domain codes as the historical converter: 0 for a clean conversion, 10 when unsupported items are preserved opaque in best-effort mode, and 20 when unsupported input is refused in strict mode. Both commands leave the source path, inode, and bytes untouched; they never rename it or create a .bak copy.

Current schema-transform support is intentionally narrow:

--resume is deliberately not crash recovery. It accepts only an output whose four default in-output contracts are complete and mutually consistent, whose source path/fingerprint, target, and migration mode match the current command, and whose checksums and output inventory still verify. It does not write any file and returns the prior terminal code (0 or 10). External contract paths are never resume authority. Missing, external, malformed, partial, or mismatching contract state is refused with code 20. Here, attested means internally consistent contracts, inventories, and checksums; it is not a cryptographic signature or protection against an actor that can rewrite every output contract and payload coherently.

For a real migration, the only contract paths allowed inside --output are the four default root files. Custom manifest, report, id-map, or unsupported report paths must be external to --output; this keeps later standalone legacy verify checks unambiguous.

--copy-only preserves bytes only when source entry names are representable in the portable slash-separated checksum ledger. In particular, a POSIX filename containing a literal backslash is refused before output creation rather than creating an output that cannot later verify.

Before --copy-only creates its private source stage, it reserves the logical size of every source file (including sparse files and each hard-link name) on the relevant TMPDIR, output, and external-contract volumes. Requests sharing a filesystem are summed because the private source and sibling publication trees coexist. A capacity shortfall or runtime ENOSPC/quota failure is a policy refusal (exit 40), not an invalid archive. The copied payload and all default in-output contracts are written to a private sibling directory and published with one directory rename; before that commit, a missing output remains missing and an existing empty output remains empty if the run fails. Custom external contracts are prepared and atomically replaced independently after that output commit, because separate filesystems cannot share one global atomic transaction.

• legacy export-n4mm can, only after explicit --trusted-load-joblib, prove a finite affine equation from exactly a fitted sklearn PLSRegression and export it through the public pls4all binding (Methods ABI 2.3) as a native PREDICT-only model.n4mm plus an attestation. It refuses pipelines and arbitrary estimators, never runs automatically, and never fabricates a workspace or archive: a standalone joblib has no signed graph, score, cohort, or lineage evidence;

• sqlite-workspace-legacy-arrays metadata is lowered to store.sqlite schema v2;

• the legacy prediction_arrays table is decoded offline, lowered to the runtime array sidecar schema (arrays/<dataset>.parquet), and also preserved in preserved/legacy-prediction-arrays.jsonl for audit;

• one standalone store.duckdb in the exact historical six-table profile is lowered to workspace-v2 metadata plus flat runtime array sidecars when both the duckdb and parquet extras are installed. It accepts a closed graph of runs, pipelines, chains, predictions, logs, and complete finite equal-length arrays; it refuses artifact references, sidecar filename collisions, unknown source files, and broken graph ownership before output creation. The original database remains checksummed under preserved/duckdb-workspace/;

• one standalone current dag-ml native-results-v1 directory with exact integer schema_version: 3, a valid score_set_hash, and canonical predictions.parquet projection is lowered to workspace-v2 run/pipeline/chain/prediction/artifact metadata plus runtime-readable arrays/<dataset>.parquet sidecars; the original native payload is still checksummed under preserved/native-results-v1/;

• malformed, older, mixed, or multi-artifact native-results-v1 sources fail --strict with a machine-checkable schema/preflight cause, and best-effort mode preserves them opaque with the same reason in the manifest;

• one standalone complete *_predictions.json loose-prediction payload is lowered to workspace-v2 run/pipeline/chain/prediction metadata plus runtime-readable arrays/<dataset>.parquet sidecars when the parquet extra is installed; the original loose JSON and sibling metadata files are still checksummed under preserved/loose-predictions/;

• one standalone legacy runs/*/*/manifest.yaml tree is lowered when its single manifest points to one complete *_predictions.json under the same source root and run_id, pipeline_id, dataset, model, and preprocessing metadata match; the manifest tree and referenced prediction payload remain checksummed under preserved/;

• a .n4a is preserved only after a bounded structural ZIP preflight: one finite JSON-object manifest.json, portable relative member names, no encrypted/symlink/special members, no normalization collisions, and explicit archive/member/expansion limits. It is copied opaque under preserved/ and is never extracted, deserialized, or made predictive; .n4a.py and non-lowerable native-results-v1 artifacts remain opaque checksummed payloads with an empty workspace-v2 store;

• non-lowerable DuckDB variants, legacy runs/ trees outside the single-manifest preview, incomplete or mixed loose prediction files, and already-v2 SQLite stores are preserved opaque by default in best-effort mode; --strict refuses them before writing;

• every real migration writes unsupported-report.json alongside the manifest, report, and id-map; dry runs write the same machine-readable unsupported report only when --unsupported-report PATH is provided;

• best-effort migration exits 10 only when semantic lowering is unavailable and content must be preserved opaque;

• --strict requires semantic lowering and exits 0 for fully lowered array sources or native-results metadata previews.

Exit codes

Contracts

Four durable JSON contracts are emitted alongside a migrated workspace (SW4_MIG_CONVERTER_spec.md §7–10):

• legacy_migration_manifest.v1 — the exhaustive inventory + checksum + id-map ledger;
• legacy_migration_report.v1 — the human/UX digest + next action;
• legacy_id_map.v1 — the never-lossy old→new id map.
• legacy_unsupported_report.v1 — the machine-readable list of unsupported, refused, or opaque-preserved items.

Development

ruff check .
mypy
pytest

Checked-in converter goldens live under tests/fixtures/legacy/. They are small reduced legacy payloads for old workspaces, run/pipeline manifests, and prediction arrays. Tests copy or materialize them into temporary directories before migration so the source goldens stay read-only and the no-in-place contract remains observable.

License

Dual-licensed CeCILL-2.1 OR AGPL-3.0-or-later (plus commercial), consistent with the nirs4all ecosystem policy. See LICENSE for the summary and LICENSES/CeCILL-2.1.txt / LICENSES/AGPL-3.0-or-later.txt for the complete canonical texts. Contact: nirs4all-admin@cirad.fr.