Skip to main content

GlobusIRI

GlobusIRI submits and monitors jobs on DOE HPC facilities through the IRI API and moves files with Globus:

  • HTTPS carries run scripts up to the HPC and brings logs back.
  • E2E Transfer moves bulk input and output data.

Supported sites: NERSC Perlmutter and ALCF Crux.

iri login (once) → iri/https/e2e refresh (cron)
→ [e2e stagein] → https upload → iri submit → iri monitor → https download logs → [e2e stageout]

Install

pip install "globusiri[panda]"
export GLOBUSIRI_WORK_DIR=/path/to/globusiri_work   # optional; default: per-user data dir
globusiri init && globusiri paths

Config, tokens, logs, and run state all live under GLOBUSIRI_WORK_DIR. If the variable is not set, they go in the per-user globusiri data directory.

Three ways to drive a workflow (same steps, one engine that never blocks):

globusiri run    --job perlmutter_example_job.yaml   # blocking: returns when the run ends
globusiri submit --job perlmutter_example_job.yaml   # start a run and return
globusiri monitor                                    # plugin mode: each call checks every active run and triggers its next step
globusiri daemon --interval 60                       # daemon mode: monitor in a loop

The job YAML describes the batch job (resources, template_file), the inputs uploaded over HTTPS (inputs), and the files moved by E2E (staging.e2e.stagein/stageout, as a manifest, Rucio DIDs, or an inline files: list).

On Perlmutter, do not call srun in the job script. NERSC's IRI already starts the script under srun, once per rank (n_node × n_process_per_node copies); a nested srun waits for CPUs the ranks hold until the job's time limit. On Crux the script runs once and starts its ranks with mpiexec. See several processes per node.

Submit as many jobs as needed, including jobs at different sites. One monitor call checks all active runs in the same WORK_DIR; daemon keeps doing that and picks up new submissions:

globusiri submit --job perlmutter_hello_job.yaml
globusiri submit --job perlmutter_wordcount_job.yaml   # first create WORK_DIR/inputs/words.txt
globusiri submit --job crux_hello_job.yaml
globusiri monitor                 # one pass over all active runs, then exits
globusiri daemon --interval 60    # continuous monitoring

Bookkeeping uses <WORK_DIR>/runs/index.db (SQLite) by default, or MySQL/MariaDB (for example the Harvester database) with database.url in sites.yaml (configuration). Full state is in runs/<run_id>/state.json. Normal monitoring queries active runs in the index; it does not scan the working directory. Concurrent submitters and monitors share file locks and database transactions; with SQLite only one process writes at a time. Use a local filesystem with POSIX locking and SQLite WAL support. The index is rebuilt from saved states when missing, or with globusiri runs --reindex.

See job examples for setup, file uploads, E2E transfers, and running many jobs with one monitor.

Python API

Every command is also a method of globusiri.GlobusIRI:

from globusiri import GlobusIRI

g = GlobusIRI()
sub = g.job_submit("perlmutter_example_job.yaml", set=["resources.n_node=2"])
state, report = g.job_wait("NERSC_Perlmutter", sub["job_id"])

r = g.submit("perlmutter_example_job.yaml")          # non-blocking workflow
g.monitor()[r["run_id"]]["plugin_status"]            # call each cycle: submitted/running/finished/failed

Documentation

Development

python -m venv .venv && .venv/bin/pip install -e ".[test]"
.venv/bin/pytest

Status: implemented (v0.1.0). The unit tests run without network access. Logins, transfers, and job submission still need to be tried against the real NERSC and ALCF services.

Metadata

Release files for globusiri 0.0.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for globusiri 0.0.3
File Size Uploaded Provenance Actions
globusiri-0.0.3.tar.gz 147.7 kB Without provenance Details for globusiri-0.0.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for globusiri 0.0.3
File Interpreter ABI Platform Actions
globusiri-0.0.3-py3-none-any.whl Python 3 none any Details for globusiri-0.0.3-py3-none-any.whl

Total release size: 255.6 kB

Release files / globusiri-0.0.3.tar.gz

Download URL globusiri-0.0.3.tar.gz
Size 147.7 kB
Tags Source
SHA-256 checksum
How to use checksums
fd940ba8026621fb5a2f96d94365d95ee0ac3c9be05685f3eb5583cc1f9ab29b
BLAKE2b-256 checksum
How to use checksums
1ad507294fe116b4f43a189b0312a3e47913abf288928f0a1819d9cf910522df
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.16

Release files / globusiri-0.0.3-py3-none-any.whl

Download URL globusiri-0.0.3-py3-none-any.whl
Size 108.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
40cae7a515f5fb7008c8d451ac32b80b56c8cdde1aaebc525b6b5cd08da5da79
BLAKE2b-256 checksum
How to use checksums
8a34d138916903c8eb0807e5403f081cfc0ee79de65c1b94db6f6b89081451e7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.16

Security for globusiri 0.0.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page