IPython magic for SLURM.
Project description
SLURM-MAGIC
This package implements magic commands for interacting with the SLURM workload manager.
SLURM magic simply wraps command-line executables and the commands themselves should look like their command-line counterparts.
Commands are spawned via subprocess
and output captured in the notebook after being decoded to UTF-8.
Whatever arguments are accepted by a SLURM command line executable are also accepted by the corresponding magic command.
Mostly you just type whatever you'd usually type at the command line just with a % in front.
Some commands are modal.
The %squeue
magic command respects a "pandas" mode meaning that the result can be returned as a Pandas dataframe.
This may not be the best way to handle this kind of functionality, and may change.
Suggestions on how to handle the output are welcome --- the raw responses are not easily readable in IPython notebooks.
Installation
slurm-magic
can be installed with pip
:
pip install slurm-magic
SLURM Magic Commands
First, things to do:
- Strategy (or no implementation) for interactive Slurm commands.
- The "mode" thing needs to be renamed.
- Logical approach to srun (auto-wrap in salloc?).
- Implement and document how to get help.
%sacct
Display accounting data for all jobs and job steps in the Slurm job accounting log or Slurm database. Is modal.
%sacctmgr
View and modify Slurm account information. Is modal.
%salloc
Obtain a Slurm job allocation (a set of nodes), execute a command, and then release the allocation when the command is finished.
%sattach (TBC)
Attach to a Slurm job step.
%%sbatch
Submit a batch script to Slurm.
This is a cell magic command that takes the contents of a cell and submits it to the batch queue.
In [2]: %%sbatch -p debug -t 10 -N 1
#!/bin/bash
srun -n 32 hostname
...:
Out[2]: u'Submitted batch job 2754280\n'
In [3]: !cat slurm-2754280.out
nid00044
nid00044
...
%sbcast (TBC)
Transmit a file to the nodes allocated to a Slurm job.
%scancel
Used to signal jobs or job steps that are under the control of Slurm.
%scontrol
Used view and modify Slurm configuration and state.
%sdiag
Scheduling diagnostic tool for Slurm. Is modal.
%sinfo
View information about Slurm nodes and partitions. Is modal.
%smap (TBC)
Graphically view information about Slurm jobs, partitions, and set configurations parameters.
%sprio
View the factors that comprise a job's scheduling priority. Is modal.
%squeue
View information about jobs located in the Slurm scheduling queue. Is modal.
To switch to "pandas" mode simply do:
In [2]: %mode pandas
Out[2]: 'pandas'
Now the output from %squeue will be a Pandas dataframe:
In [3]: %squeue -u rthomas
Out[3]:
JOBID USER ACCOUNT NAME PARTITION QOS NODES TIME_LIMIT TIME ST \
0 2764292 rthomas mpccc sh debug debug 1 10:00 0:11 R
PRIORITY SUBMIT_TIME START_TIME
0 69060 2016-07-21T21:12:34 2016-07-21T21:13:21
%sreport (TBC)
Generate reports from the slurm accounting data.
%srun
Run parallel jobs.
%sshare
Tool for listing the shares of associations to a cluster. Is modal.
%sstat (TBC)
Display various status information of a running job/step.
%strigger (TBC)
Used set, get or clear Slurm trigger information.
%sview (TBC)
Graphical user interface to view and modify Slurm state.
What Our Users Say
I'll never have to leave a notebook again
that's like the ultimate dream
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.