fwl-io

Shared data-download utilities for the PROTEUS ecosystem: manifest-driven, mirrored, offline-first fetching of the reference data used by the FormingWorlds models.

Status: version 0, released for inspection and feedback by the collaboration. The API is not yet frozen; suggestions are welcome as issues. Documentation: proteus-framework.org/fwl-io.

What it does

Models in the PROTEUS framework depend on reference data hosted on Zenodo (with Dataverse mirrors): spectral files, equation-of-state lookup tables, stellar evolution tracks. fwl-io provides one shared mechanism to declare, verify, and fetch these datasets into the common FWL_DATA directory tree:

• Manifests: each dataset is declared in a TOML manifest with its location below FWL_DATA, its Zenodo version DOI, an optional Dataverse mirror DOI, and the models that require it. fwl-io ships a manifest for datasets shared across models; each model package can ship its own manifest (plus the generated registry files, both included in its package data) and expose it through the fwl_io.manifests entry-point group, so adding data to a model never requires an fwl-io release. A provider whose manifest fails to load is skipped with a warning; it cannot break data access for the other models.
• Committed registries: the file names and checksums of every dataset live in registry files generated by fwl-io sync from the Zenodo API. Hashes are pinned in version control and reviewed like code; nothing is hand-edited. The checksums are the ones Zenodo publishes (md5 today), so the Dataverse mirror must host byte-identical copies of the originals; disable Dataverse's tabular ingest for mirrored deposits.
• Version DOIs only: manifests pin the DOI of a specific deposit, and every dataset requires a Zenodo version DOI (Dataverse serves as a download mirror). fwl-io sync rejects concept DOIs, which resolve to the newest deposit and would let data drift underneath pinned code.
• Mirrored downloads: every file is fetched with pooch, verified against its checksum, tried against Zenodo first and the Dataverse mirror second, and moved into place atomically (staged on the same filesystem, os.replace into place) so an interrupted download can never leave a corrupt file behind. Local placement failures such as a read-only tree or a full disk are reported as themselves, never as mirror failures. A deposit packaged as a single archive (extract = "tar" / "zip") is downloaded, checksum-verified, and unpacked into the dataset directory, with the archive itself discarded.
• Concurrent-safe: when many processes start together and all need the same missing file, a per-file inter-process lock lets only one download it while the others wait and reuse the result, so a batch of runs cannot stampede Zenodo/Dataverse into rate-limiting the collaboration. The lock is coherent across cluster nodes where the shared filesystem supports it, and best-effort: if locking is unavailable or a holder stalls, waiters fall back to fetching unguarded rather than failing.
• Offline-first: with FWL_IO_OFFLINE=1 no network access is attempted; files resolve from the local tree or fail with an actionable error. A read-only, pre-populated group cache (FWL_DATA_CACHE) is consulted before any download, which serves cluster nodes without internet access.

Installation

pip install fwl-io

The distribution and the command are named fwl-io; the Python import is fwl_io, since Python does not allow hyphens in module names.

Usage

from fwl_io import create_fetcher, fetch_for

eos = create_fetcher(
    subdir='interior/eos/mgsio3_demo',
    zenodo='10.5281/zenodo.1234567',
    dataverse='10.34894/ABCDEF',
    registry='interior.eos.mgsio3_demo.registry.txt',
)
path = eos.fetch('density.dat')      # cached, verified, atomic
paths = fetch_for('aragog')          # everything a model requires
records = eos.provenance()           # (file, source, checksum) for run manifests

Command line:

fwl-io list                     # datasets from all installed manifests
fwl-io fetch aragog             # fetch everything aragog requires
fwl-io sync manifest.toml       # regenerate committed registries from Zenodo

Environment variables

Adding a dataset

1. Upload the files to Zenodo and note the version DOI of the deposit.
2. Declare the dataset in the appropriate manifest: the shared manifest in this repository for data consumed by several models, or the consuming model's own manifest for model-specific data.
3. Run fwl-io sync <manifest> and commit the manifest change together with the generated registry file. Packages that ship their own manifest must include both the manifest and its registry files in their package data, or fetching fails at runtime on the user's machine.
4. Optionally mirror the deposit to Dataverse (byte-identical, tabular ingest disabled) and add the dataverse DOI.

Development

pip install -e ".[develop]"
pytest -m "unit or smoke"       # fast tier, no network
pytest                          # full suite; uses a local test server only
ruff check src tests && ruff format --check src tests

The test suite never contacts external services; download logic is exercised against a local HTTP server.