GlobusIRI
GlobusIRI submits and monitors jobs on DOE HPC facilities through the IRI API and moves files with Globus:
- HTTPS carries run scripts up to the HPC and brings logs back.
- E2E Transfer moves bulk input and output data.
Supported sites: NERSC Perlmutter and ALCF Crux.
iri login (once) → iri/https/e2e refresh (cron)
→ [e2e stagein] → https upload → iri submit → iri monitor → https download logs → [e2e stageout]
Install
pip install "globusiri[panda]"
export GLOBUSIRI_WORK_DIR=/path/to/globusiri_work # optional; default: per-user data dir
globusiri init && globusiri paths
Config, tokens, logs, and run state all live under GLOBUSIRI_WORK_DIR. If the variable is
not set, they go in the per-user globusiri data directory.
Three ways to drive a workflow (same steps, one engine that never blocks):
globusiri run --job perlmutter_example_job.yaml # blocking: returns when the run ends
globusiri submit --job perlmutter_example_job.yaml # start a run and return
globusiri monitor # plugin mode: each call checks every active run and triggers its next step
globusiri daemon --interval 60 # daemon mode: monitor in a loop
The job YAML describes the batch job (resources, template_file), the inputs uploaded over
HTTPS (inputs), and the files moved by E2E (staging.e2e.stagein/stageout, as a manifest,
Rucio DIDs, or an inline files: list).
On Perlmutter, do not call
srunin the job script. NERSC's IRI already starts the script undersrun, once per rank (n_node × n_process_per_nodecopies); a nestedsrunwaits for CPUs the ranks hold until the job's time limit. On Crux the script runs once and starts its ranks withmpiexec. See several processes per node.
Submit as many jobs as needed, including jobs at different sites. One monitor call checks
all active runs in the same WORK_DIR; daemon keeps doing that and picks up new submissions:
globusiri submit --job perlmutter_hello_job.yaml
globusiri submit --job perlmutter_wordcount_job.yaml # first create WORK_DIR/inputs/words.txt
globusiri submit --job crux_hello_job.yaml
globusiri monitor # one pass over all active runs, then exits
globusiri daemon --interval 60 # continuous monitoring
Bookkeeping uses <WORK_DIR>/runs/index.db (SQLite) by default, or MySQL/MariaDB (for
example the Harvester database) with database.url in sites.yaml
(configuration). Full state is in
runs/<run_id>/state.json. Normal monitoring queries active runs in the index; it does not
scan the working directory. Concurrent submitters and monitors share file locks and database
transactions; with SQLite only one process writes at a time. Use a local filesystem with
POSIX locking and SQLite WAL support. The index
is rebuilt from saved states when missing, or with globusiri runs --reindex.
See job examples for setup, file uploads, E2E transfers, and running many jobs with one monitor.
Python API
Every command is also a method of globusiri.GlobusIRI:
from globusiri import GlobusIRI
g = GlobusIRI()
sub = g.job_submit("perlmutter_example_job.yaml", set=["resources.n_node=2"])
state, report = g.job_wait("NERSC_Perlmutter", sub["job_id"])
r = g.submit("perlmutter_example_job.yaml") # non-blocking workflow
g.monitor()[r["run_id"]]["plugin_status"] # call each cycle: submitted/running/finished/failed
Documentation
- docs/design.md: packaging and local layout, architecture, token model, workflow, and open questions
- docs/configuration.md: WORK_DIR and the
sites.yamlschema - docs/cli.md:
globusiricommands and cron setup - docs/api.md: the Python API (
globusiri.GlobusIRI), one method per command - docs/examples.md: shell jobs, local inputs, E2E transfers, and many-job monitoring
- src/globusiri/data/sites.example.yaml: example site config for
NERSC_PerlmutterandALCF_CRUX(fixed per site) - src/globusiri/data/jobs/: example job files, copied into WORK_DIR by
globusiri init
Development
python -m venv .venv && .venv/bin/pip install -e ".[test]"
.venv/bin/pytest
Status: implemented (v0.1.0). The unit tests run without network access. Logins, transfers, and job submission still need to be tried against the real NERSC and ALCF services.
Metadata
Release files for globusiri 0.0.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| globusiri-0.0.3.tar.gz | 147.7 kB | Without provenance | Details for globusiri-0.0.3.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| globusiri-0.0.3-py3-none-any.whl | Python 3 | none | any | Details for globusiri-0.0.3-py3-none-any.whl |
Total release size: 255.6 kB
Release files / globusiri-0.0.3.tar.gz
| Download URL | globusiri-0.0.3.tar.gz |
|---|---|
| Size | 147.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
fd940ba8026621fb5a2f96d94365d95ee0ac3c9be05685f3eb5583cc1f9ab29b
|
|
BLAKE2b-256 checksum How to use checksums |
1ad507294fe116b4f43a189b0312a3e47913abf288928f0a1819d9cf910522df
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.16
|
Release files / globusiri-0.0.3-py3-none-any.whl
| Download URL | globusiri-0.0.3-py3-none-any.whl |
|---|---|
| Size | 108.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
40cae7a515f5fb7008c8d451ac32b80b56c8cdde1aaebc525b6b5cd08da5da79
|
|
BLAKE2b-256 checksum How to use checksums |
8a34d138916903c8eb0807e5403f081cfc0ee79de65c1b94db6f6b89081451e7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.16
|
Security for globusiri 0.0.3
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.