Foldspace

Foldspace allows to schedule and run web services on compute nodes of a remote HPC clusters and access them from your local browser. The backend services are mainly (but not necessarily) run via Apptainer/Singularity containers.

The main inspiration for Foldspace is Open Ondemand, from which it inherits the main logics as well as part of the codes that it uses on the remote side; Foldspace replaces the Apache + nginx role in Open Ondemand with Python scripts + ssh tunnels, in order to be daemonless. In this way, any user can run backend services (tools) from containers in a remote directory (toolbox) with no needs of servers running on the HPC cluster (other than sshd).

Install

[!note] Foldspace is officially supported on GNU/Linux, MacOS and Windows via WSL. Windows Powershell support is still experimental, and BSD systems have not been tested.

You can install the latest Foldspace release in a Python virtual environment¹ of your current path using for instance:

python3 -m venv foldspace_venv # or choose the name you like
source foldspace_venv/bin/activate
pip install --upgrade pip
pip install foldspace

If you prefer to use the development version instead, clone the official repository and replace pip install foldspace with pip install <pip_package directory path>.

Quick start

A typical general syntax for Foldspace is the following:

foldspace --user <my username on cluster> --host <url or host in ~/.ssh/config for login nodes> --toolbox <remote path with tools containers> --tool <name of the tool to start> [plus any option for the scheduler submit command]

but for default clusters like e.g. Cineca clusters you can exploit some default settings and ready-to-use toolboxes on the clusters to make this syntax shorter. For instance, to use Foldspace on Leonardo you can run something like this:

foldspace --user <my username on cluster> --cluster leonardo --tool <name of the tool to start> [any sbatch option you like]

You can list all the default HPC clusters using the --list-clusters option and the avaliable tools remotely with the --list-tools option. Please note that using the default clusters might imply to automatically run some script on your local computer to allow the access to the specific cluster². For instance, in the case of Cineca clusters, a script will run to retrieve Smallstep certificates and to set ssh known hosts properly for login nodes, in order to ease the access to the cluster via ssh; if you don't like this behaviour you can disable it via the --no-startup-script option and perform those steps manually before using Foldspace.

Here it follows a list of examples to make it clearer:

# Run Jupyter on Leonardo with default configs (1 task, 1 hour on lrd_all_serial partition, no gpus, using your default account)
foldspace --user <my user name on leonardo> --cluster leonardo --tool jupyter-latest

# Run Jupyter on Leonardo for 3 hours using 4 tasks and one gpu on boost_usr_prod partition with your account <my account>
foldspace --user <my user name on leonardo> --cluster leonardo --tool jupyter-latest --ntasks 4 --gres gpu:1 --partition boost_usr_prod --account <my account> --time 03:00:00

# Run a VNC desktop via noVNC + TurboVNC  on Leonardo with default parameters (1 task, 1 hour on boost_usr_prod partition, 1 gpu, using your default account)
foldspace --user <my user name on leonardo> --cluster leonardo --tool vnc

# Run a VNC desktop via noVNC + TurboVNC  on Leonardo using the boost_qos_dbg for 30 minutes
foldspace --user <my user name on leonardo> --cluster leonardo --tool vnc -q boost_qos_dbg --time 30:00

Please check the --help option for the full list of options, as well as the documentation of your scheduler³ and your cluster⁴ for the full set of job submit options.

Official repository

Foldspace source code is hosted at this link, where you can also find documentation and examples related to the toolbox to be hosted remotely.

Custom toolbox

If you want to build your toolbox with your own tools you can exploit the --toolbox option; if coupled with --cluster it overwrites the default system value, whereas if you are deploying Foldspace on a different HPC cluster you need to couple it with the --host option.

Toolboxes has to be built with a specific structure in order to be compliant with the client package here. Please refer to the original repo for details (see toolboxes/README.md).

License

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see here.

Mantainers

This code is mantained by HPC department at Cineca.

Authors

Fabio Pitari, Lucia Rodriguez Muñoz, Antonio Memmolo