scherbelberg 0.0.6

HPC cluster deployment and management for the Hetzner Cloud

SCHERBELBERG

/ˈʃɛʁbɛlbɛʁk/ (German, toponym, male: "mountain of shards" in Leipzig)

Synopsis

scherbelberg provides both a CLI and an API for deploying and managing small Dask-based HPC clusters in the Hetzner cloud. Development status alpha, stability acceptable, security also acceptable but implementation needs a review.

Project's Name

Next to impressive projects like Fugaku, which is named after Mount Fuji, the TOP500 are clearly missing an entry from the city of Leipzig. This project is named after one of the few significant "mountains" in the city, the "Scherbelberg", also known as the "Rosentalhügel": 20 meters above the surrounding landscape and 125 meters above sea level. Starting out as a late 19th century landfill, it has since become part of a park-like landscape. As of 1975, a famously shaky steel observation tower with a rather beautiful view is located at its top, overlooking not only the Leipziger Auenwald forest but also the city's sewage treatment plant.

Installation

scherbelberg can be installed via conda as follows:

conda install -c conda-forge scherbelberg

This package has been tested on Linux and Windows 10. It should work on most Unix-like systems. You must run a conda environment based entirely on recent versions of conda-forge packages with CPython versions 3.8, 3.9 or 3.10. ssh must be installed separately as a prerequisite. A Hetzner API token is required. By default, scherbelberg expects it to be located in the HETZNER environment variable.

See chapter on installation in scherbelberg's documentation for further details. Also see section on how to get started for additional steps.

CLI

Similar to git, the CLI is divided into sub-commands. They all come with their own parameters. Information on the latter can be found by using the --help option.

~> scherbelberg --help
Usage: scherbelberg [OPTIONS] COMMAND [ARGS]...

  HPC cluster deployment and management for the Hetzner Cloud

Options:
  --version
  --help     Show this message and exit.

Commands:
  catalog  list data centers and available servers types
  create   create cluster
  destroy  destroy cluster
  ls       list cluster nodes
  nuke     nuke cluster
  scp      scp from/to cluster node
  ssh      ssh into cluster node

See chapter on CLI in scherbelberg's documentation for further details.

API

scherbelberg uses asyncio. A cluster can basically be created and destroyed, with or without asyncio:

from asyncio import run
from scherbelberg import Cluster

c = await Cluster.from_new(**kwargs)
# or
c = run(Cluster.from_new(**kwargs))

await c.destroy()
# or
run(c.destroy())

Access to an earlier established cluster can also be gained:

from asyncio import run
from scherbelberg import Cluster

c = await Cluster.from_existing(**kwargs)
# or
c = run(Cluster.from_existing(**kwargs))

Once the cluster has been created or gained access to, one can simply request an initialized Dask client object:

dask_client = await c.get_client()
# or
dask_client = run(c.get_client(asynchronous = False))

See chapter on API in scherbelberg's documentation for further details.