oscar-benchmarking 0.2.0

A package for submitting benchmarking scripts on OSCAR.

SlurmJobSubmitter Python Package

This Python package submits jobs to a Slurm scheduler.

How to Use This Package and Its Modules

This package sources its parameters from a YAML configuration file. Please follow the structure outlined under YAML: General Job Configuration. This package also uses a CSV configuration file that stores run-to-run configuration. Please follow the structure outlined under CSV: Run-Specific Parameters.

This package has two modules that users will often interact with: SlurmJobSubmitter and SlurmScriptWriter. SlurmScriptWriter automatically forms the script body needed to run the specified benchmark. SlurmJobSubmitter writes script content formed by a SlurmScriptWriter to file and submits the script to the Slurm scheduler.

SlurmJobSubmitter

This module writes script content formed by a SlurmScriptWriter to file and submits the script to the Slurm scheduler.

Classes:

SlurmJobSubmitter()

This class is an abstract base class for submitting jobs to the Slurm scheduler.

MLPerfJobSubmitter()

This class, derived from SlurmJobSubmitter, submits MLPerf jobs to the Slurm scheduler.

NOTE: This class only submits jobs from ONE script. If you need to write another script, declare another submitter object.

Arguments:

• script_writer (SlurmScriptWriter): The script writer to use.
• script_path (str): File path to write to.
• nodes (int): The number of nodes to use.
• gres (str): Generic resources required for the job.
• ntasks_per_node (int): Number of tasks to run per node.
• memory (str): Amount of memory required.
• time (str): Maximum time the job can run.
• partition (str): The partition to submit the job to.
• error_file_name (str, optional): Name of the file to which standard error will be written. Defaults to "%j.err".
• output_file_name (str, optional): Name of the file to which standard output will be written. Defaults to "%j.out".
• account (str, optional): Account to be charged for the job. Defaults to None.
• num_runs (int, optional): Number of times to run the MLPerf script. Defaults to 1.

Functions:

write()

Writes to a batch script to file, given a file path and script content.

Raises OSError, if opening the file fails.

submit()

Submits the SBATCH script file to gpu node(s).

SlurmScriptWriter

Classes:

SlurmScriptWriter()

This abstract base class is a template for script-writing classes that writes the body of Slurm scripts.

MLPerfScriptWriter()

This class, derived from SlurmScriptWriter, writes the script body for MLPerf scripts.

Arguments:

• run_id (int): Unique ID of this benchmark run
• benchmark (str): Type of benchmark
• model (str): Name of model (e.g. resnet50, bert99, etc.)
• backend (str): Backend of the model (e.g. tf, torch)
• arch (str): GPU architecture (e.g. gh200, h100, etc.)
• container_image (str): Path to pre-configured Apptainer image
• data_path (str): Path to dataset
• model_config (dict): Configuration for the model. Keys are model args, values are their values. Model args may be different for different models.

Functions:

config_logger()

This function configures the logger and creates its log file.

Returns: The Logger object.

generate_script_body()

This function will create the body of the sbatch file.

Raises LookupError, if the CM commands for the model are not found.

slurm_input()

Read input from slurm commands like sfeature and get the actual list of gpus to submit to.

Returns: The list of GPU IDs.

Config Files

YAML: General Job Configuration

Configure the job parameters you need for the specific MLPerf job in config.yaml. You can set configurations for several model, benchmark, backend, and architecture combinations. You need to specify:

• SBATCH parameters
• Path to the Apptainer image container
• CM-command parameters
• Path to the dataset

The following diagram is the structure of the config.yaml file.

# General script parameters

# Architecture-specific parameters
arch: 
    arch-1:
        # SBATCH parameters
        param-1:
        param-2:
        ...

        # Apptainer image path
        container_image:
    arch-2:

# Model-specific parameters
model:
    resnet50:
        # CM parameters
        cm-param-1:
        cm-param-2:
        ...

        # Path to dataset
        data_path: 

Example

Here is an example of a valid YAML configuration.

# General script parameters
destination: "./"
num_runs: 1

# Architecture-specific parameters
arch:
  arm64-gracehopper: &arch_config
    # SBATCH parameters
    nodes: 1
    partition: "gracehopper"
    gres: "gpu:1"
    account: "ccv-gh200-gcondo"
    ntasks_per_node: 1
    memory: "40G"
    time: "01:00:00"
    error_file_name: "%j.err"
    output_file_name: "%j.out"

    # Apptainer image path
    container_image: "/oscar/data/shared/eval_gracehopper/container_images/MLPerf/arm64/mlperf-resnet-50-tf-arm64"

# Model-specific CM parameters
model:
  resnet50: &model_config
    # CM parameters
    hw_name: "default"
    implementation: "reference"
    device: "cuda"
    scenario: "Offline"
    adr.compiler.tags: "gcc"
    target_qps: 1
    category: "edge"
    division: "open"

    # Path to dataset
    data_path: "/oscar/data/ccvinter/mstu/gracehopper_eval/data/imagenet/ILSVRC2012/val"
  bert-99: 
    <<: *model_config
    # CM parameters inherited from model_config template
    # Overrides the dataset path
    data_path: "/oscar/data/ccvinter/mstu/gracehopper_eval/data/imagenet/ILSVRC2012/val"

The &model_config declares a template called model_config. You can reference this template for additional models (e.g. bert-99) by replacing the ampersand & with an asterisk (*). To override any values in the template, you need to use <<: before *model_config.

CSV: Run-Specific Parameters

You can set run-specific parameters (run ID, model, benchmark, backend, architecture, gpu node) for each MLPerf benchmark configuration you want to run.

RUN_ID,BENCHMARK,MODEL,BACKEND,ARCH,NODE
1,MLPerf-Inference,resnet50,tf,arm64-gracehopper,gpu2701

Running MLPerf Job as an Array Job

Set a variable num_runs in config.yaml to run several of the same benchmark on the same GPU node. This variable will submit an array job to Slurm and aggregate your results across all runs.

Developers

New Classes

If you are developing to add features for a new kind of Slurm job, you should write a derived class from the ABC for both script generation and job submitting.

Writing a Python Submit Script Using This Package

See submit_resnet-50.py and submit_bert-99.py for examples.

You may use one of the functions in util.py to get the parameters from the config files. from_yaml_config() will typically be used when running exactly 1 run configuration. from_yaml_csv_config() will be used if you need to specify several run configs in a CSV file.

Utilizing the Logger

You can grab the Logger object associated with this job by accessing the MLPerfJobSubmitter object's Logger attribute. Then, you can log to your heart's content!