Skip to main content

GlobusIRI

GlobusIRI submits and monitors jobs on DOE HPC facilities through the IRI API and moves files with Globus:

  • HTTPS carries run scripts up to the HPC and brings logs back.
  • E2E Transfer moves bulk input and output data.

Supported sites: NERSC Perlmutter and ALCF Crux.

iri login (once) → iri/https/e2e refresh (cron)
→ [e2e stagein] → https upload → iri submit → iri monitor → https download logs → [e2e stageout]

Install

pip install "globusiri[panda]"
export GLOBUSIRI_WORK_DIR=/path/to/globusiri_work   # optional; default: per-user data dir
globusiri init && globusiri paths

Config, tokens, logs, and run state all live under GLOBUSIRI_WORK_DIR. If the variable is not set, they go in the per-user globusiri data directory.

The generated config/sites.yaml lists two example sites (NERSC_Perlmutter, ALCF_CRUX). Edit it to work on your own sites: add sites (more NERSC or ALCF resources, other projects or work directories) or remove the ones you do not use. globusiri init never overwrites it. See adding and removing sites.

Three ways to drive a workflow (same steps, one engine that never blocks):

globusiri run    --job perlmutter_example_job.yaml   # blocking: returns when the run ends
globusiri submit --job perlmutter_example_job.yaml   # start a run and return
globusiri monitor                                    # plugin mode: each call checks every active run and triggers its next step
globusiri daemon --interval 60                       # daemon mode: monitor in a loop

The job YAML describes the batch job (resources, template_file), the inputs uploaded over HTTPS (inputs), and the files moved by E2E (staging.e2e.stagein/stageout, as a manifest, Rucio DIDs, or an inline files: list).

On Perlmutter, do not call srun in the job script. NERSC's IRI already starts the script under srun, once per rank (n_node × n_process_per_node copies); a nested srun waits for CPUs the ranks hold until the job's time limit. On Crux the script runs once and starts its ranks with mpiexec. See several processes per node.

Submit as many jobs as needed, including jobs at different sites. One monitor call checks all active runs in the same WORK_DIR; daemon keeps doing that and picks up new submissions:

globusiri submit --job perlmutter_hello_job.yaml
globusiri submit --job perlmutter_wordcount_job.yaml   # first create WORK_DIR/inputs/words.txt
globusiri submit --job crux_hello_job.yaml
globusiri monitor                 # one pass over all active runs, then exits
globusiri daemon --interval 60    # continuous monitoring

Bookkeeping uses <WORK_DIR>/runs/index.db (SQLite) by default, or MySQL/MariaDB (for example the Harvester database) with database.url in sites.yaml (configuration). Full state is in runs/<run_id>/state.json. Normal monitoring queries active runs in the index; it does not scan the working directory. Concurrent submitters and monitors share file locks and database transactions; with SQLite only one process writes at a time. Use a local filesystem with POSIX locking and SQLite WAL support. The index is rebuilt from saved states when missing, or with globusiri runs --reindex.

See job examples for setup, file uploads, E2E transfers, and running many jobs with one monitor.

Python API

Every command is also a method of globusiri.GlobusIRI:

from globusiri import GlobusIRI

g = GlobusIRI()
sub = g.job_submit("perlmutter_example_job.yaml", set=["resources.n_node=2"])
state, report = g.job_wait("NERSC_Perlmutter", sub["job_id"])

r = g.submit("perlmutter_example_job.yaml")          # non-blocking workflow
g.monitor()[r["run_id"]]["plugin_status"]            # call each cycle: submitted/running/finished/failed

Documentation

Development

python -m venv .venv && .venv/bin/pip install -e ".[test]"
.venv/bin/pytest

Status: implemented (v0.1.0). The unit tests run without network access. Logins, transfers, and job submission still need to be tried against the real NERSC and ALCF services.

Metadata

Release files for globusiri 0.0.6

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for globusiri 0.0.6
File Size Uploaded Provenance Actions
globusiri-0.0.6.tar.gz 162.7 kB Without provenance Details for globusiri-0.0.6.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for globusiri 0.0.6
File Interpreter ABI Platform Actions
globusiri-0.0.6-py3-none-any.whl Python 3 none any Details for globusiri-0.0.6-py3-none-any.whl

Total release size: 281.7 kB

Release files / globusiri-0.0.6.tar.gz

Download URL globusiri-0.0.6.tar.gz
Size 162.7 kB
Tags Source
SHA-256 checksum
How to use checksums
130665346604d81e0f5ce273917527773d35bc3662b480284f5319018d61674d
BLAKE2b-256 checksum
How to use checksums
7ed6e4a6ad45e8a5b4074f6f51ab0a033f9f0fde5d2ff58d6fbcb89e193c6a2d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.16

Release files / globusiri-0.0.6-py3-none-any.whl

Download URL globusiri-0.0.6-py3-none-any.whl
Size 119.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
11e65031c2b012e8215e60239c70fbd0e6a42cffeba39a8f24d84d1547bab07a
BLAKE2b-256 checksum
How to use checksums
aeeb72c0ae05c3febd1f101011308205ab6acd990acbf285369991744e7ff0d9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.16

Security for globusiri 0.0.6

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page