cloudmesh-gpu 5.0.3

A cloudmesh command called gpu that can be used to monitor the GPU during runtime

Cloudmesh Command gpu

Note: This file is automatically created. Please do not modify it. Please change the code instead.

  Usage:
        gpu watch [--gpu=GPU] [--delay=SECONDS] [--logfile=LOGFILE] [--count=COUNT] [--dense]
        gpu --json [--gpu=GPU] [--pretty] [FILE]
        gpu --xml
        gpu --yaml
        gpu ps [--gpu=GPU] [--format=FORMAT] [--detail]
        gpu system
        gpu status
        gpu count
        gpu kill
        gpu show --output=OUTPUT FILE [--plot=PLOT] [--frequency=FREQUENCY]
        gpu probe
        gpu

  This command returns some information about NVIDIA GPUs if your
  system has them. Please use the cms command in the commandline with
  `cms gpu`

  Options:
      --json                 returns the information in json
      --xml                  returns the information in xml
      --yaml                 returns the information in xml
      --logfile=LOGFILE      the logfile
      --count=COUNT          how many times the watch is run [default: -1]
      --dense                do not print any spaces [default: False]
      --detail               short process names [default: False]
      --format=FORMAT        table, json, yaml [default: table]
      --plot=PLOT            timeseries, histogram [default: timeseries]
      --frequency=FREQUENCY  absolute, percent [default: percent]
      --gpu=GPUS             which graphics cards

  Description:

    Although some GPU information tools exist, we did not find all information
    that we needed. Also, the output format was not convenient enough for later
    analysis. The program we developed can obtain information at specified
    intervals. Note that at this time the tool is restricted to NVIDIA GPUs.

        cms gpu kill
            In case you run the gpu command in the background you can kill it with the
            `kill command`

        cms gpu count
            Returns the number of GPUS.

        gpu watch [--gpu=GPU] [--delay=SECONDS] [--logfile=LOGFILE] [--count=COUNT] [--dense]
            This command allows to print environmental variables in a continuous fashion.
            If count is used the command is executed that number of times. If it is omitted
            it runs continiously. The delay specifies how many seconds between invocations are
            dealyed. The `logfile` specifies an output file in which the events are written.
            By default spaces are included so the output has a table like format. The spaces
            can be eliminated with dense so that less space is used in the log file.
            In case multiple GPUs are present one can select specific GPUs for which the
            monitoring entries are generated.
            The watch command output is proceeded by a header that is adapted based on the
            gpus specified. This makes it possible to read in the file as dataframe and apply
            easily plotting tools.

            An example output looks like (the newline in this example in the header was
            introduced to increase readability of this documentation. In an execution it is
            one line.

            # ####################################################################################
            # time, 0 id, 0 gpu_util %, 0 memory_util %, 0 encoder_util %, 0 decoder_util %,
            # ... 0 gpu_temp C, 0 power_draw W
            2022-03-18 11:26:40.877006,   0,  11,  16,   0,   0,    49,    47.47
            2022-03-18 11:26:40.983229,   0,  17,  17,   0,   0,    49,    47.59
            2022-03-18 11:26:41.093406,   0,  17,  17,   0,   0,    49,    47.88

        gpu --json [--gpu=GPU] [--pretty] [FILE]
            Prints out the information in json format. We have eliminated some of the attributes
            that are not important to us at this time. with the `pretty` flag the json is printed
            in indented pretty format.
            The FILE parameter is used to read in a saved instance from nvidia-smi.

        gpu --xml
            Prints out the information in xml format. This is the format that is retrieved from
            nvidia-smi. The gpu selection flag is not enabled for this format. If you want to
            implement it, create a pull request.

        gpu --yaml
            Prints out the information in yaml format. The gpu selection flag is not enabled
            for this format. If you want to implement it, create a pull request.

        gpu ps [--gpu=GPU] [--format=FORMAT] [--detail]
            Prints out the processes running on the GPU in the specified format for the selected GPUs.
            The process name is shortened based on removing the path of the command. If the full path
            is needed one can use the `detail` flag. Allowed formats are table, csv, json, and yaml.

            A sample output for a table looks similar to

        +-----+-----+-----+------+-------------+---------------------+-----------------+--------------+
        | job | gpu | pid | type | used_memory | compute_instance_id | gpu_instance_id | process_name |
        +-----+-----+-----+------+-------------+---------------------+-----------------+--------------+
        | 1   | 0   | 173 | G    | 198 MiB     | N/A                 | N/A             | Xorg         |
        | 2   | 0   | 260 | G    | 610 MiB     | N/A                 | N/A             | Xorg         |
        | 3   | 0   | 274 | G    | 49 MiB      | N/A                 | N/A             | gnome-shell  |
        | 4   | 0   | 436 | G    | 27 MiB      | N/A                 | N/A             | zoom         |
        | 5   | 0   | 321 | G    | 100 MiB     | N/A                 | N/A             | slack        |
        | 6   | 0   | 591 | G    | 11 MiB      | N/A                 | N/A             | firefox      |
        +-----+-----+-----+------+-------------+---------------------+-----------------+--------------+


        gpu system
            Returns information about the GPU card

                [
                  {
                    "product_name": "NVIDIA GeForce RTX 3090",
                    "product_brand": "GeForce",
                    "product_architecture": "Ampere",
                    "vbios_version": "94.02.42.00.A9",
                    "inforom_version": {
                      "img_version": "G001.0000.03.03",
                      "oem_object": "2.0",
                      "ecc_object": "N/A",
                      "pwr_object": "N/A"
                    },
                    "accounted_processes": null,
                    "vendor": "ASUSTeK Computer Inc. Device [abcd:1234]"
                  }
                ]


        gpu status
            Returns elementary status information such as environmental sensors

                [
                  {
                    "fan_speed": "0 %",
                    "utilization": {
                      "gpu_util": "18 %",
                      "memory_util": "18 %",
                      "encoder_util": "0 %",
                      "decoder_util": "0 %"
                    },
                    "temperature": {
                      "gpu_temp": "48 C",
                      "gpu_temp_max_threshold": "98 C",
                      "gpu_temp_slow_threshold": "95 C",
                      "gpu_temp_max_gpu_threshold": "93 C",
                      "gpu_target_temperature": "83 C",
                      "memory_temp": "N/A",
                      "gpu_temp_max_mem_threshold": "N/A"
                    },
                    "supported_gpu_target_temp": {
                      "gpu_target_temp_min": "65 C",
                      "gpu_target_temp_max": "91 C"
                    },
                    "power_readings": {
                      "power_state": "P8",
                      "power_management": "Supported",
                      "power_draw": "47.28 W",
                      "power_limit": "390.00 W",
                      "default_power_limit": "390.00 W",
                      "enforced_power_limit": "390.00 W",
                      "min_power_limit": "100.00 W",
                      "max_power_limit": "480.00 W"
                    },
                    "clocks": {
                      "graphics_clock": "210 MHz",
                      "sm_clock": "210 MHz",
                      "mem_clock": "405 MHz",
                      "video_clock": "555 MHz"
                    },
                    "voltage": {
                      "graphics_volt": "737.500 mV"
                    },
                    "accounted_processes": null
                  }
                ]