slurmise

Table of Contents

• Installation
• License

Installation

pip install slurmise

Usage

Docs

Visit the docs for more information.

Configuration

Slurmise requires a configuration file for every command which tells slurmise where to store important files and how to process input files. A general CLI parser is hard to get correct and still doesn't account for all use cases, for example is the input file size important or number of lines? In addition to some built-in file parsers, you have the option of utilizing awk to extract more complex information from a file.

Example toml

[slurmise]
# base directory to store database and optimized models
base_dir = "slurmise_dir"

# how much the database may grow past the records a model was fit on before
# `predict` warns that the model should be refit, as a fraction of those records.
# Default is 0.2, so a model fit on 20 jobs warns once the database holds 25.
retrain_warning_threshold = 0.2

# set to false to silence the refit warning entirely.  Default is true.
retrain_warning_enable = true

# Global resource bounds applied to all jobs unless overridden per-job.
# Built-in defaults: 60 minutes for runtime, 1000 MB for memory.
[slurmise.runtime]
default = 70          # minutes returned when no model is available
minimum = 10          # never predict below this
maximum = 1440        # never predict above this (e.g. 24-hour wall-time limit)
multiply_prediction_by = 1.0   # scale factor applied after prediction
retry_exponent = 1.0  # on Snakemake retry attempt N: prediction *= N**retry_exponent
on_high_uncertainty_return = "prediction"  # "default", "prediction", "max", or "min"

[slurmise.memory]
default = 2000        # MB returned when no model is available
minimum = 500
maximum = 500000
multiply_prediction_by = 1.0
retry_exponent = 1.0
on_high_uncertainty_return = "prediction"

# Per-job overrides: any field from [slurmise.runtime] / [slurmise.memory] can be
# specified here and will override the global value for that job only.
[slurmise.job.job_name.runtime]
default = 80          # override global default for this job

[slurmise.job.job_name.memory]
default = 3000

# the variables section defines types and how to parse the command
[slurmise.job.job_name.variables]
# the job spec determines how to parse commands to extract their relevant,
# independent variables
threads = {type = "numeric"}
complexity = {type = "category"}

The job_spec key can be added to define how to parse a command string; its placeholders reference variable names defined in the variables section:

[slurmise.job.job_name]
job_spec = "subcommand -T {threads} -C {complexity}"

[slurmise.job.job_name.variables]
threads = {type = "numeric"}
complexity = {type = "category"}

Matching job names

The job name of a command can be set in various ways. First, if the command starts with the job name, the job name will be detected and removed from the command.

# slurmise.toml
[slurmise.job.git]
job_spec = "checkout {branch}"

[slurmise.job.git.variables]
branch = {type = "category"}

This job specification will match any of the following with the branch set to my_branch

# infer from command
slurmise record "git checkout my_branch"
# tell explicitly
slurmise record --job-name git "checkout my_branch"

Since the job name cannot have spaces and some commands have several subcommands, you can set unique prefixes for certain jobs.

# slurmise.toml
[slurmise.job.git_checkout]
job_prefix = "git checkout"
job_spec = "{branch}"

[slurmise.job.git_checkout.variables]
branch = {type = "category"}

[slurmise.job.git_merge]
job_prefix = "git merge"
job_spec = "{branch}"

[slurmise.job.git_merge.variables]
branch = {type = "category"}

With a job prefix, slurmise will use the prefix instead of job name to infer job names.

slurmise record "git checkout my_branch"
# explicit job name
slurmise record -j git_checkout "my_branch"

Note that the job name and prefix should not be included in the job specification. When a job name is explicitly given to slurmise, the corresponding command should not have the prefix or job name included.

Job specifications

Variables are defined in the [slurmise.job.job_name.variables] section. Each variable requires a type:

[slurmise.job.job_name.variables]
variable_name = {type = "variable_type"}

The name should be unique within a job and contain no spaces. The type can be one of:

• numeric: A single number, used in regression as an independent variable. Examples include the number of threads, epochs, or replicates to perform. Integer, decimal, and scientific notation values (e.g. 1e-4, 2.5E+6) are all supported.
• category: A string which is used to select the correct model. Examples include what algorithm to choose, switches or flags. Note that a category can be a number, but will be stored as a string, e.g. "1.0" is different from "1". For inference, the categories will be matched to particular, independent model.
• file: An input file in plain text. Can be processed further as described below.
• gzip_file: An input file in gzip format. During processing, the file will be decompressed to read it's contents, note this can incur memory and cpu drain.
• file_list: An input file that contains a list of files to process in turn.

When a job_spec is provided, its placeholders reference the variable names:

{variable_name}

By default each placeholder matches a single whitespace-delimited token. The special token {ignore} can appear in job_spec to match and discard one token that is not recorded; it does not require a variable name or entry in the variables section. Use multiple {ignore} tokens to skip multiple arguments:

job_spec = "cmd {input} {ignore} {ignore} {output}"

Matching multi-word or non-standard tokens

When a placeholder needs to match something other than a single whitespace-delimited word — for example, multiple arguments to discard, a file path containing spaces, or a value delimited by something other than whitespace — add a pattern key to the variable entry with a Python regular expression:

[slurmise.job.my_job.variables]
# absorb a variable number of extra arguments at the end of the command
extra_args = {type = "ignore", pattern = ".+"}

# file path wrapped in double quotes (e.g. cmd "/path/with spaces/file.txt")
input = {type = "file", file_parsers = "file_basename", pattern = '[^"]+'}

For the quoted-file case, surround the placeholder with literal quote characters in job_spec so the quotes are consumed but not captured:

job_spec = 'cmd "{input}" {output}'

The pattern key is not supported for numeric variables, whose matching pattern is fixed to ensure the captured value can always be converted to a number.

To include a literal { or } in a job spec (for example in an awk program or a shell brace expression), double the brace: {{ produces a literal { and }} produces a literal }.

job_spec = "awk '{{print $1}}' {input}"

This matches commands like awk '{print $1}' data.txt and captures data.txt as the input variable.

File Parsers

Each file can have one or more parsers associated with its variable name. Slurmise comes with several built-in options for parsing files:

• file_size: The size of the file on disk, in bytes, numeric
• file_lines: The number of lines (newlines) in the file, numeric
• file_basename: The base filename, category
• file_md5: The md5 digest of the file contents, category

Additionally, custom file parsers can be made using awk. While somewhat limited, awk prevents security issues with running arbitrary code. File parsers require a unique name in the slurmise.file_parsers collection. The return type is category by default. The awk command can be supplied as a string or file path, which is used with the -f flag of awk. Here are some examples:

[slurmise.file_parsers.epochs]
type = "numeric"
awk_script = "/^epochs:/ {print $2}"

[slurmise.file_parsers.network]
type = "category"
awk_script = "/^network type:/ {print $3}"

[slurmise.file_parsers.fasta_length]
type = "numeric"
awk_script = "/path/to/awk/file.awk"
script_is_file = True

# contents of file.awk
# /^>/ {if (seq) print seq; seq=0}
# /^>/ {next}
# {seq = seq + length($0)}
# END {if (seq) print seq}

The first extracts the token after epochs: as a number and could be used for getting metadata from a configuration file. Similarly, the network parser extracts the network type but this time returns the result as a category.

Finally, the fasta_length parser takes an awk script file that prints the length of each sequence in a fasta file, returning the list of numbers as numerics.

To specify which parser a file uses, add a file_parsers key to the variable entry:

[slurmise.job.sample_files]
job_spec = "--reference {reference} {fasta}"

[slurmise.job.sample_files.variables]
reference = {type = "file", file_parsers = ["file_md5", "file_lines"]}
fasta = {type = "file", file_parsers = ["file_size", "fasta_length"]}

Each file variable needs a file_parsers entry. The value can be a single string or a list of parsers. Here the reference file is parsed with the md5 and number of lines. The md5 will create a new category based on the file contents while the lines will be an independent variable. In practice, matching md5 will ensure the same number of lines which doesn't provide additional information to the model. The fasta file returns the file size in bytes and the number of nucleotides in each fasta entry.

License

slurmise is distributed under the terms of the MIT license.