remotemanager
Modular serialisation and management package for handling the running of functions on remote machines
Based off of the BigDFT RemoteRunner concept, remotemanager represents an improvement and expansion on the concepts based there.
Primary usage is via a Dataset, which connects to a remote machine via URL
You can think of the Dataset as a "container" of sorts for a calculation, to which "runs" are attached. These runs are then executed on the remote machine described by the provided URL
Installation
A quick install of the latest stable release can be done via pip install remotemanager
For development, you can clone this repo and install via cd remotemanager && pip install -e .[dev]
Tip: You can clone a specific branch with git clone -b devel.
If you want to build the docs locally a pandoc install is required.
You can install all required python packages with the [dev] or [docs] optionals.
HPC
Remotemanager exists to facilitate running on High Performance Compute machines (supercomputers). Script generation is ideally done via the BaseComputer module.
Existing Computers can be found at this repository. For creating a new machine class, see the documentation.
Documentation
See the documentation for further information, tutorials and api documentation.
Quickstart
This section will run through running a very basic function on a machine.
It roughly echoes the quickstart page found in the docs.
Function Definition
Start by defining your "calculation" as a python function.
def multiply(a, b):
import time
time.sleep(1)
return a * b
Remote Connection
We need to be able to connect to a remote machine to run this function.
Assuming you can connect to a machine with a string like ssh user@host or just ssh machine, you can directly create
a connection.
Use the URL module for this.
from remotemanager import URL
connection = URL("ssh@machine")
Commands
You can execute commands on this machine using the cmd method:
connection.cmd("pwd")
>> > "/home/user"
Running Functions
To execute your function on the specified machine, create a Dataset
from remotemanager import Dataset
ds = Dataset(function=multiply, url=connection, name="test")
Adding Runs
You can specify the inputs that your function should use by adding Runner instances to the Dataset
ds.append_run({"a": 10, "b": 7})
Running
Now run your Dataset with run().
You can wait for completion with wait()
ds.run()
ds.wait(1, 10)
Here, we are waiting for a maximum of 10s, and checking for results every 1s
Results
Result collection is done in two stages.
Once a run is complete, we must first fetch the results with fetch_results.
Then we can access the results at the results property
ds.fetch_results()
ds.results
>> > [70]
sanzu
The sanzu functionality allows you to tag a jupyter cell for remote running.
The cell will be converted into a function, set off to the specified remote machine, and executed there.
For detailed information, see the relevant section of the docs
To use this functionality, first enable the magic:
%load_ext
remotemanager
You should then create a URL instance for your machine:
from remotemanager import URL
connection = URL(...)
And now we can execute any cell on this machine by using the %%sanzu protocol:
%%sanzu
url = connection
%%sargs
a = 10
%%sargs
b = 7
a * b
>> > 70
This can be useful for doing file operations on the remote machine, however it is possible to access your results.
Accessing Sanzu Results
A sanzu run will inject a magic_dataset object into the jupyter runtime.
This is the Dataset that was used to execute the most recent cell, so you can access the information from there.
For our last run, we can see our results here:
print(magic_dataset.results)
>> > [70]
Release files for remotemanager 0.14.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| remotemanager-0.14.3.tar.gz | 109.4 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| remotemanager-0.14.3-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 237.4 kB
Release files / remotemanager-0.14.3.tar.gz
| Download URL | remotemanager-0.14.3.tar.gz |
|---|---|
| Size | 109.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
46d68054dae910e97456f50b420d3c68181bcfa56d38e1e24262b99f743fd04d
|
|
BLAKE2b-256 checksum How to use checksums |
9afbce04cd1ec8c85c4b2290d76f5df951aa815ecad767ffd1fc6344cf073dd1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.10.8
|
Release files / remotemanager-0.14.3-py3-none-any.whl
| Download URL | remotemanager-0.14.3-py3-none-any.whl |
|---|---|
| Size | 128.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
0c81244eb10268b2475eedc81301484c7aa95f7f06d5c690485163335981ac08
|
|
BLAKE2b-256 checksum How to use checksums |
05fc93d1960e9906a3c9210b13d6aeff831622a99c945530c79b9947626a9069
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.10.8
|
Security for remotemanager 0.14.3
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
