multiverse_cache

Download, cache, and read the reference resources used by the M-PACT pipeline. Three kinds of resource are managed through one machine-level cache:

• genomes — genome "info" packages (from kylessmith/<genome>_info on GitHub)
• datasets — Zenodo data records (e.g. methylation annotation parquet tables)
• models — Zenodo model records (e.g. M-PACT .pth checkpoints)
• sturgeon — Sturgeon CNS classifier models (zip archives, from the Sturgeon project)

Nothing is written into the installed package directory. Everything lands in a per-machine cache created on first use and reused thereafter.

Cache location

Resolved in priority order:

1. an explicit cache_dir= argument
2. the MULTIVERSE_CACHE environment variable
3. $XDG_CACHE_HOME/multiverse_cache
4. ~/.cache/multiverse_cache

On a cluster, point it at scratch to avoid $HOME quotas:

export MULTIVERSE_CACHE=/scratch/$USER/multiverse_cache

Layout:

<root>/
  base/<genome>.pickle           # genome info manifest
  data/<genome>/...              # genome info data + external/
  datasets/<name>/...            # Zenodo datasets (e.g. methyl_anno)
  models/<name>/...              # Zenodo models (e.g. MPACT)
  sturgeon/<name>/<model>.zip    # Sturgeon models (kept zipped, never unpacked)
  <name>/.complete               # per-resource completion marker
  <key>.lock                     # advisory download lock

Usage

Genome info (unchanged API)

from multiverse_cache import InfoReader
info = InfoReader("hg38")        # downloads once, cached after
frame = info["some_key"]

Datasets (replaces import_data / download_data)

from multiverse_cache import get_data_file, download_dataset, list_data_files

download_dataset("methyl_anno")              # explicit prefetch (optional)
path = get_data_file("some.parquet")         # auto-downloads methyl_anno, returns path
files = list_data_files("methyl_anno")

get_data_file(filename) is the cache-aware drop-in for the old function: it ensures the dataset is present, then locates the file inside the cache. Pass dataset=None to search across all cached datasets without downloading.

Models (new)

from multiverse_cache import download_models, list_models, get_model_file, load_model

download_models("MPACT")                     # explicit prefetch (optional)
list_models("MPACT")                         # -> [".../MPACT_classifier.pth", ...]
ckpt = load_model("MPACT_classifier.pth")    # torch.load(map_location="cpu") -> object
path = get_model_file("MPACT*.pth")          # just the path, no load

load_model lazily imports torch and defaults to map_location="cpu" so a checkpoint trained on H100s loads on a login node before you move it to a device. weights_only defaults to False (trusted first-party checkpoints that bundle config/objects); set True for untrusted files.

Sturgeon CNS models (new)

Sturgeon (Vermeulen, Pagès-Gallego, Kester et al., Nature 2023) ships each CNS classifier as a zip holding an ONNX network plus probe/decoding/calibration tables. The zip is the unit of distribution and is opened as-is by Sturgeon's own loader, so it is cached intact and never unpacked.

Licensing. Sturgeon is distributed under an Evaluation Software License Agreement (Oncode/Cyclomics/UMCU), not an open-source licence: academic research only, no derivative works / reimplementation, and clinical/diagnostic use needs a separate commercial licence (software@cyclomics.com); publishing results requires referral to the source. This package therefore drives the genuine Sturgeon package — it does not vendor or reimplement Sturgeon's prediction code.

from multiverse_cache import (
    list_sturgeon, download_sturgeon, get_sturgeon_model_file, predict_sturgeon,
)

list_sturgeon()                              # -> ['brainstem', 'general']

# Path-only access (no extra deps): hand this to the sturgeon CLI / loader
zip_path = get_sturgeon_model_file("general")

Predicting (recommended: isolated, zero env impact)

Sturgeon's released code needs pandas<2.2, numpy<2, an old onnxruntime, and matplotlib — versions that usually clash with a modern torch/Mamba stack. predict_sturgeon sidesteps that by building a dedicated venv under the cache (once), installing the genuine Sturgeon + compatible deps into it, and running Sturgeon's own predict CLI there. Your main environment is never modified.

# bed = output of `sturgeon inputtobed` (T2T/CHM13v2-aligned methylation calls)
scores = predict_sturgeon("sample.bed", model="general")   # first call builds the venv
# -> {"sample": <per-class score DataFrame>}

Interpreter requirement. Those pinned deps only ship wheels for Python 3.9–3.12. The venv is built from such an interpreter, resolved in order: python_executable= / $MULTIVERSE_STURGEON_PYTHON → the current interpreter if it qualifies → the newest python3.X on PATH → a conda/mamba env.

So if your session is Python 3.13/3.14 (e.g. a bleeding-edge mamba env) and no python3.X is on PATH, predict_sturgeon will, by default, reuse a conda env named sturgeon or create one automatically:

mamba create -y -n sturgeon python=3.10      # run for you (mamba/micromamba/conda)

This assumes a conda/mamba tool is installed (it picks mamba → micromamba → conda). Controls:

• predict_sturgeon(..., auto_conda=False) disables the auto-create (you'll get a clear error instead);
• sturgeon_env_python(conda_env=..., conda_python=...) change the env name / Python version;
• python_executable=... or export MULTIVERSE_STURGEON_PYTHON=... pin a specific interpreter and skip discovery entirely (recommended for reproducible cluster runs).

(Without a compatible interpreter, pip would try to compile pandas 2.1.x from source and fail against the newer CPython C-API — which is the error you hit on a 3.14 env.)

sturgeon_env_python(python_executable=...) builds/returns that venv's interpreter directly, and predict_sturgeon(..., env_python=...) lets you point at a conda env where you already have a compatible Sturgeon.

In-process loading (only on an already-compatible env)

If this environment already satisfies Sturgeon's pins, you can load the ONNX session in-process:

from multiverse_cache import load_sturgeon, install_sturgeon

install_sturgeon()                  # pip-installs genuine sturgeon into THIS env
model  = load_sturgeon("general")   # or load_sturgeon("general", auto_install=True)
df     = model.predict("sample.bed")

install_sturgeon() installs the package unmodified with --no-deps --ignore-requires-python (so it won't downgrade your numpy/pandas), but prediction will still fail here if this env has pandas>=2.2 / numpy>=2 — hence the isolated route above is preferred. download_sturgeon(..., install=True) fetches the model and installs Sturgeon in one call.

The models are hosted on Dropbox; ?dl=0 share links are normalised to ?dl=1 automatically. If those links rot, re-register the resource with new files (see below).

To add a Sturgeon model from your own URL:

from multiverse_cache import register, Resource, STURGEON
register(Resource(name="sturgeon_custom", category=STURGEON, source="url",
                  extract=False,
                  files=(("custom.zip", "https://host/custom.zip"),)))
predict_sturgeon("sample.bed", model="custom")

MARLIN acute-leukemia classifier (Python/ONNX, no R)

MARLIN (Steinicke, Benfatto et al., Nature Genetics 2025, MIT-licensed) classifies acute leukemia from sparse methylation profiles with a Keras network. MARLIN ships an R prediction script, but this plugin needs no R: the trained model is converted from Keras HDF5 to ONNX once (in a throwaway TensorFlow env that is then discarded) and run with onnxruntime, exactly like the Sturgeon runtime. The reference probe order (an .RData) is read with pyreadr and the class table (.xlsx) with pandas.

from multiverse_cache import predict_marlin

# bed: chrom  start  end  methylation(0-1 or NA)  probe   (e.g. modkit pileup
#      intersected with the MARLIN probe coordinates)
scores = predict_marlin("sample.bed")     # first call: download + one-time convert
# -> DataFrame: samples (rows) x 42 methylation classes (cols), softmax probs

Runtime dependencies are pure Python: pip install "multiverse_cache[marlin]" (onnxruntime, pyreadr, openpyxl). The one-time HDF5→ONNX conversion needs TensorFlow + tf2onnx; rather than touch your environment, it builds a temporary venv from a 3.9–3.11 interpreter (TF 2.13's wheel range), converts, caches marlin_v1.model.onnx, and deletes the temp env. If your session is outside 3.9–3.11 it uses the same conda/mamba fallback as Sturgeon (predict_marlin(..., auto_conda=...), python_executable=..., or $MULTIVERSE_MARLIN_PYTHON). Pass keep_convert_env=True to keep the TF env for re-conversion.

Acquisition splits across sources automatically: the model HDF5 from Zenodo (marlin_model) and the probe/annotation files from GitHub (marlin_refs).

from multiverse_cache import download_marlin, load_marlin
download_marlin()                 # prefetch model + refs (optional)
model = load_marlin()             # convert-if-needed + onnxruntime session
df = model.predict("beds/")       # a directory of beds, one sample each

Registering more resources

from multiverse_cache import register, Resource, MODELS, DATASETS

register(Resource(name="MPACT_v2", category=MODELS, record_id="12345678",
                  description="next M-PACT checkpoints"))
load_model("MPACT_v2_classifier.pth", name="MPACT_v2")

Managing the cache

from multiverse_cache import MultiverseCache
cache = MultiverseCache("/scratch/me/multiverse_cache")
cache.list_genomes()                 # downloaded genome-info packages
cache.list_resources("models")       # downloaded models
cache.list_resources("datasets")     # downloaded datasets
cache.remove_resource("models", "MPACT")
cache.clear()

Behavior notes

• Atomic downloads — every file streams to a .part temp and is renamed into place only on success; checksums from Zenodo are verified before rename.
• Completion markers — a resource is only "ready" once .complete exists, so a half-finished download is re-fetched rather than read.
• Concurrency — downloads take a per-resource flock, so an LSF/SLURM array all requesting the same genome/dataset/model fetches it once; the rest wait.
• Extension-aware extraction — archives are unpacked by extension (.zip, .tar, .tar.gz/.tgz, .gz); everything else (.parquet, .pth, .npz, ...) is left intact. This fixes a bug in the original loader, which force-extracted every download as a zip and would corrupt a torch .pth checkpoint (those are themselves zip archives). A resource can also set extract=False to keep its archives whole even when they end in .zip — used for Sturgeon model zips, which their own loader opens with zipfile.ZipFile.
• Multiple sources — a resource's source is either "zenodo" (files, sizes and checksums discovered from a record) or "url" (explicit (filename, url) pairs, e.g. Dropbox-hosted Sturgeon models).
• Source accounts/records: genome info from REPO_USER in download.py; dataset/model records in registry.py.