modchef

Cook EasyBuild environments from software ingredients.

modchef indexes a cluster's EasyBuild modules into an RDF graph and turns a list of requested tools and Python/R packages into a minimal, ready-to-run set of module load commands — picking versions that share a compatible toolchain. It distinguishes modules that are installed (loadable now) from those only available in the official EasyBuild collection, and when something isn't installed it tells you the exact easyconfig to request. When requested tools can't share a toolchain, it can suggest the install that would unify them.

See docs/user-guide.md for the end-user guide.

Usage

modchef cook --tools samtools bcftools multiqc --python pandas numpy
modchef cook recipe.yaml --output load_modules.sh
modchef search bwa
modchef explain SAMtools/1.22-GCC-14.3.0
modchef menu

cook emits a module purge followed by the module load lines for the fewest compatible toolchain clusters. It also surfaces, as comments and on stderr: REQUEST INSTALL (available in EasyBuild, not installed), TO UNIFY (install X so split tools share one toolchain), and NOT IN EASYBUILD (no easyconfig anywhere). search tags each hit [installed] or [available — not installed].

Daily index (cron, runs as the EasyBuild admin user)

modchef-index needs the EasyBuild framework importable, so the cron loads the EasyBuild module first (the runtime modchef module deliberately does not depend on EasyBuild, so cook/search users stay lean):

module load EasyBuild/5.2.0
modchef-index --installed-root /opt/easybuild/software \
              --robot-repo     /opt/easybuild/easyconfigs \
              --official-repo  /opt/easybuild/easyconfigs \
              --output /opt/easybuild/modchef/modchef.ttl

--installed-root is the EasyBuild software install tree: modchef indexes the easyconfig EasyBuild stamped for each actually installed module (<name>/<version>/easybuild/*.eb), so the catalog matches what module avail shows, including every bundle's extensions. --robot-repo is an easyconfigs collection used only to resolve installed dependencies. --official-repo is the official EasyBuild collection, indexed as "available, not installed" with the same full facts (deps + packages): cook builds from installed modules first and, when a tool or package is only available, tells you the easyconfig to ask support to install. Any easyconfig that fails to parse is reported on stderr rather than silently dropped. Full-parsing the official collection makes the daily index noticeably heavier.

The runtime CLI reads the graph from $MODCHEF_TTL (set by the module file).

Deploying the module (EasyBuild, on the HPC as the EasyBuild admin user)

Two easyconfigs ship in easybuild/, differing only in toolchain and dependency versions:

Pick the one whose Python matches the rest of your interactive tooling — loading modchef beside a module built on a different Python puts two Pythons in one environment, and which one wins depends on load order.

Both need two site-specific edits before installing:

• modextravars = {'MODCHEF_TTL': ...} — where the modchef-index cron writes the daily graph. Users must be able to read it, your EasyBuild admin to write it.
• checksum — the tarball is built from a checkout, so its bytes differ per build and no published value can be correct for you. Generate yours with eb <easyconfig> --inject-checksums. Once modchef is on PyPI you can instead set source_urls = [PYPI_SOURCE] and pin the released hash.

The installed module is named after the toolchain, e.g. modchef/1.0.0-GCCcore-15.2.0.

1. Build the source tarball from a checkout and copy it plus your chosen easyconfig to the HPC (<eb> below is the toolchain suffix you picked):

    python -m build                   # writes dist/modchef-1.0.0.tar.gz
    scp dist/modchef-1.0.0.tar.gz              <admin>@<hpc>:/opt/easybuild/ebfiles_repo/
    scp easybuild/modchef-1.0.0-<eb>.eb        <admin>@<hpc>:/opt/easybuild/ebfiles_repo/
   

2. EasyBuild caches sources by filename, so overwrite any stale copy in its sourcepath (eb --show-config | grep sourcepath) before reinstalling:

    cp /opt/easybuild/ebfiles_repo/modchef-1.0.0.tar.gz <sourcepath>/m/modchef/
   

3. Reinstall over the existing module and smoke-test:

    eb /opt/easybuild/ebfiles_repo/modchef-1.0.0-<eb>.eb --rebuild
    module load modchef/1.0.0-<eb>
    modchef --help
    modchef-index --help | grep official-repo
   

The version stays 1.0.0 (rebuild in place), so use --rebuild. An earlier SYSTEM-toolchain modchef/1.0.0, if any, is a separate module name — remove it to avoid confusion.