Skip to main content

GlobusIRI

GlobusIRI submits and monitors jobs on DOE HPC facilities through the IRI API and moves files with Globus:

  • HTTPS carries run scripts up to the HPC and brings logs back.
  • E2E Transfer moves bulk input and output data.

Supported sites: NERSC Perlmutter and ALCF Crux.

iri login (once) → iri/https/e2e refresh (cron)
→ [e2e stagein] → https upload → iri submit → iri monitor → https download logs → [e2e stageout]

Install

pip install "globusiri[panda]"
export GLOBUSIRI_WORK_DIR=/path/to/globusiri_work   # optional; default: per-user data dir
globusiri init && globusiri paths

Config, tokens, logs, and run state all live under GLOBUSIRI_WORK_DIR. If the variable is not set, they go in the per-user globusiri data directory.

The generated config/sites.yaml lists two example sites (NERSC_Perlmutter, ALCF_CRUX). Edit it to work on your own sites: add sites (more NERSC or ALCF resources, other projects or work directories) or remove the ones you do not use. globusiri init never overwrites it. See adding and removing sites.

Three ways to drive a workflow (same steps, one engine that never blocks):

globusiri run    --job perlmutter_example_job.yaml   # blocking: returns when the run ends
globusiri submit --job perlmutter_example_job.yaml   # start a run and return
globusiri monitor                                    # plugin mode: each call checks every active run and triggers its next step
globusiri daemon --interval 60                       # daemon mode: monitor in a loop

The job YAML describes the batch job (resources, template_file), the inputs uploaded over HTTPS (inputs), and the files moved by E2E (staging.e2e.stagein/stageout, as a manifest, Rucio DIDs, or an inline files: list).

On Perlmutter, do not call srun in the job script. NERSC's IRI already starts the script under srun, once per rank (n_node × n_process_per_node copies); a nested srun waits for CPUs the ranks hold until the job's time limit. On Crux the script runs once and starts its ranks with mpiexec. See several processes per node.

Submit as many jobs as needed, including jobs at different sites. One monitor call checks all active runs in the same WORK_DIR; daemon keeps doing that and picks up new submissions:

globusiri submit --job perlmutter_hello_job.yaml
globusiri submit --job perlmutter_wordcount_job.yaml   # first create WORK_DIR/inputs/words.txt
globusiri submit --job crux_hello_job.yaml
globusiri monitor                 # one pass over all active runs, then exits
globusiri daemon --interval 60    # continuous monitoring

Bookkeeping uses <WORK_DIR>/runs/index.db (SQLite) by default, or MySQL/MariaDB (for example the Harvester database) with database.url in sites.yaml (configuration). Full state is in runs/<run_id>/state.json. Normal monitoring queries active runs in the index; it does not scan the working directory. Concurrent submitters and monitors share file locks and database transactions; with SQLite only one process writes at a time. Use a local filesystem with POSIX locking and SQLite WAL support. The index is rebuilt from saved states when missing, or with globusiri runs --reindex.

See job examples for setup, file uploads, E2E transfers, and running many jobs with one monitor.

Python API

Every command is also a method of globusiri.GlobusIRI:

from globusiri import GlobusIRI

g = GlobusIRI()
sub = g.job_submit("perlmutter_example_job.yaml", set=["resources.n_node=2"])
state, report = g.job_wait("NERSC_Perlmutter", sub["job_id"])

r = g.submit("perlmutter_example_job.yaml")          # non-blocking workflow
g.monitor()[r["run_id"]]["plugin_status"]            # call each cycle: submitted/running/finished/failed

Documentation

Development

python -m venv .venv && .venv/bin/pip install -e ".[test]"
.venv/bin/pytest

Status: implemented (v0.1.0). The unit tests run without network access. Logins, transfers, and job submission still need to be tried against the real NERSC and ALCF services.

Metadata

Release files for globusiri 0.0.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for globusiri 0.0.4
File Size Uploaded Provenance Actions
globusiri-0.0.4.tar.gz 157.8 kB Without provenance Details for globusiri-0.0.4.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for globusiri 0.0.4
File Interpreter ABI Platform Actions
globusiri-0.0.4-py3-none-any.whl Python 3 none any Details for globusiri-0.0.4-py3-none-any.whl

Total release size: 269.4 kB

Release files / globusiri-0.0.4.tar.gz

Download URL globusiri-0.0.4.tar.gz
Size 157.8 kB
Tags Source
SHA-256 checksum
How to use checksums
251431af3d0d8338bae62701c32a7805662e17e69ba42f0960d3577511e27e32
BLAKE2b-256 checksum
How to use checksums
4620b377d9faaf5d8545c24abe2e7600153a3cc1fe2370df934a47d4a32ceb77
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.16

Release files / globusiri-0.0.4-py3-none-any.whl

Download URL globusiri-0.0.4-py3-none-any.whl
Size 111.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
2a57e3553da6274f352a7496c98825dbe7e881456d39033a96350a7655d88b53
BLAKE2b-256 checksum
How to use checksums
22ebe33802f795585f8380fc88227c8102ab0219b9f45791c84d3f5cfa89b17a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.16

Security for globusiri 0.0.4

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page