Foldspace

Foldspace allows to schedule and run web services on compute nodes of a remote HPC clusters and access them from your local browser. The backend services are mainly (but not necessarily) run via Apptainer/Singularity containers.

The main inspiration for Foldspace is Open Ondemand, from which it inherits the main logics as well as part of the codes that it uses on the remote side; Foldspace replaces the Apache + nginx role in Open Ondemand with Python scripts + ssh tunnels, in order to be daemonless. In this way, any user can run backend services (tools) from containers in a remote directory (toolbox) with no needs of servers running on the HPC cluster (other than sshd).

Install

💡Note: Foldspace is officially supported on GNU/Linux, MacOS and Windows via WSL. Windows Powershell support is still experimental, and BSD systems have not been tested.

You can install the latest Foldspace release in a Python virtual environment¹ of your current path using for instance:

python3 -m venv foldspace_venv # or choose the name you like
source foldspace_venv/bin/activate
pip install --upgrade pip
pip install foldspace

If you prefer to use the development version instead, clone the official repository and replace pip install foldspace with pip install <pip_package directory path>².

Quick start

A typical general syntax for Foldspace is the following:

foldspace --user <my username on cluster> --host <url or host in ~/.ssh/config for login nodes> --toolbox <remote path containing the tools> --tool <name of the tool to start> [plus any option for the scheduler submit command]

but for some default clusters you can exploit some default settings and ready-to-use toolboxes on the clusters to make this syntax shorter. You can list all the default HPC clusters using the --list-clusters option and the avaliable tools remotely with the --list-tools option. For instance, to use Foldspace on <my cluster> you can run something like this:

foldspace --user <my username on cluster> --cluster <my cluster> --tool <name of the tool to start> [any sbatch option you like]

Please note that using the default clusters might imply to automatically run some script on your local computer to allow the access to the specific cluster³ (e.g. to retrieve some ssh certificate needed for the access); if you don't like this behaviour you can skipt this step via the --no-startup-script option and perform any preliminary required step manually before using Foldspace.

Here it follows a list of examples to make it clearer; all the examples refers to a generic <my cluster> using Slurm as scheduler, containing jupyter and vnc as tools in the default cluster toolbox:

# Run Jupyter on `<my cluster>` with default configs (e.g. default partition, default walltime, default account, default resources)
foldspace --user <my user name on my cluster> --cluster <my cluster> --tool jupyter

# Run Jupyter on `<my cluster>` for 3 hours using 4 tasks on `<some partition>` with your account <my account>:
foldspace --user <my user name on my cluster> --cluster <my cluster> --tool jupyter --ntasks 4 --partition <some partition> --account <my account> --time 03:00:00

# Run a VNC remote desktop on `<my cluster>` with default parameters: 
foldspace --user <my user name on my cluster> --cluster <my cluster> --tool vnc

# Run a VNC remote desktop on `<my cluster>` using <some qos> for 30 minutes
foldspace --user <my user name on my cluster> --cluster <my cluster> --tool vnc --qos <some qos> --time 30:00

Please check the --help option for the full list of options, as well as the documentation of your scheduler⁴ and your cluster for the full set of job submit options.

Official repository

Foldspace source code is hosted at this link, where you can also find documentation and examples related to the toolbox to be hosted remotely.

Custom toolbox

If you want to build your toolbox with your own tools you can exploit the --toolbox option; if coupled with --cluster it overwrites the default system value, whereas if you are deploying Foldspace on a different HPC cluster you need to couple it with the --host option.

Toolboxes has to be built with a specific structure in order to be compliant with the client package here. Please refer to the original repo for details (see toolboxes/README.md).

License

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see here.

Mantainers

This code is mantained by HPC department at Cineca.

Authors

Fabio Pitari, Lucia Rodriguez Muñoz, Antonio Memmolo