Skip to main content

Starting a jupyter kernel using srun

Project description

Slurm Jupyter Kernel

Create jupyter kernels and run kernels using srun

Jupyter Notebook/JupyterHub should have direct access to the compute nodes, so it is a good idea to install JupyterHub on a front end of the HPC system.

If you are an HPC user, you can also install the Python packages notebook and slurm_jupyter_kernel with the prefix --user into your home directory.

slurmkernel is able to connect to a kernel started on a compute node using SSH port forwarding. To allow users to access the compute node without a password, the following PAM module should have been configured:

https://slurm.schedmd.com/pam_slurm_adopt.html

Installation

Install using pip

python3 -m pip install slurm_jupyter_kernel

Create a new kernel

If you do not specify a keyfile with --keyfile it will automatically create a SSH keyfile in {home}/.ssh/{username}_jupslurm at start time. The public part will be append to the authorized_keys file.

Get help

$ slurmkernel --help

usage: Adding jupyter kernels using slurm [-h] {create} ...

positional arguments:
  {create}
    create    create a new slurm kernel

optional arguments:
  -h, --help  show this help message and exit

Example

$ slurmkernel create --displayname="Python 3.8.2" --account=hpc-group --time=00:30:00 --kernel-cmd="python3 -m ipykernel_launcher -f {connection_file}" --partition=batch

Try to create new jupyter slurm kernel "Python 3.8.2" ...
{
  "argv": [
    "/usr/bin/python3",
    "-m",
    "slurm_jupyter_kernel",
    "--partition",
    "batch",
    "--account",
    "hpc-group",
    "--time",
    "00:30:00",
    "--kernel-cmd",
    "python3 -m ipykernel_launcher -f {connection_file}",
    "--connection-file",
    "{connection_file}"
  ],
  "display_name": "SLURM Python 3.8.2"
}

Successfully created kernel "SLURM Python 3.8.2" :-)

Example

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

slurm_jupyter_kernel-1.0.tar.gz (6.2 kB view hashes)

Uploaded Source

Built Distribution

slurm_jupyter_kernel-1.0-py3-none-any.whl (7.0 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page