amd-node-scraper 1.1.0

A framework for automated error detection and data collection

Node Scraper

Node Scraper is a tool which performs automated data collection and analysis for the purposes of system debug.

Table of Contents

• Installation
  • Install From Source
• CLI Usage
  • Execution Methods
    • Example: Remote Execution
    • Example: connection_config.json
  • Subcommands
    • 'describe' subcommand
    • 'run-plugins' sub command
    • 'gen-plugin-config' sub command
    • 'compare-runs' subcommand
    • 'summary' sub command
• Configs
  • Global args
  • Plugin config: --plugin-configs command
  • Reference config: gen-reference-config command
• Extending Node Scraper (integration & external plugins) → See EXTENDING.md
• Full view of the plugins with the associated collectors & analyzers as well as the commands invoked by collectors -> See docs/PLUGIN_DOC.md

Installation

Install From Source

Node Scraper requires Python 3.9+ for installation. After cloning this repository, call dev-setup.sh script with 'source'. This script creates an editable install of Node Scraper in a python virtual environment and also configures the pre-commit hooks for the project.

source dev-setup.sh

Alternatively, follow these manual steps:

1. Virtual Environment (Optional)

python3 -m venv venv
source venv/bin/activate

On Debian/Ubuntu, you may need: sudo apt install python3-venv

2. Install from Source (Required)

python3 -m pip install --editable .[dev] --upgrade

This installs Node Scraper in editable mode with development dependencies. To verify: node-scraper --help

3. Git Hooks (Optional)

pre-commit install

Sets up pre-commit hooks for code quality checks. On Debian/Ubuntu, you may need: sudo apt install pre-commit

CLI Usage

The Node Scraper CLI can be used to run Node Scraper plugins on a target system. The following CLI options are available:

usage: node-scraper [-h] [--sys-name STRING] [--sys-location {LOCAL,REMOTE}] [--sys-interaction-level {PASSIVE,INTERACTIVE,DISRUPTIVE}] [--sys-sku STRING]
                    [--sys-platform STRING] [--plugin-configs [STRING ...]] [--system-config STRING] [--connection-config STRING] [--log-path STRING]
                    [--log-level {CRITICAL,FATAL,ERROR,WARN,WARNING,INFO,DEBUG,NOTSET}] [--gen-reference-config] [--skip-sudo]
                    {summary,run-plugins,describe,gen-plugin-config} ...

node scraper CLI

positional arguments:
  {summary,run-plugins,describe,gen-plugin-config}
                        Subcommands
    summary             Generates summary csv file
    run-plugins         Run a series of plugins
    describe            Display details on a built-in config or plugin
    gen-plugin-config   Generate a config for a plugin or list of plugins

options:
  -h, --help            show this help message and exit
  --sys-name STRING     System name (default: <my_system_name>)
  --sys-location {LOCAL,REMOTE}
                        Location of target system (default: LOCAL)
  --sys-interaction-level {PASSIVE,INTERACTIVE,DISRUPTIVE}
                        Specify system interaction level, used to determine the type of actions that plugins can perform (default: INTERACTIVE)
  --sys-sku STRING      Manually specify SKU of system (default: None)
  --sys-platform STRING
                        Specify system platform (default: None)
  --plugin-configs [STRING ...]
                        built-in config names or paths to plugin config JSONs. Available built-in configs: NodeStatus (default: None)
  --system-config STRING
                        Path to system config json (default: None)
  --connection-config STRING
                        Path to connection config json (default: None)
  --log-path STRING     Specifies local path for node scraper logs, use 'None' to disable logging (default: .)
  --log-level {CRITICAL,FATAL,ERROR,WARN,WARNING,INFO,DEBUG,NOTSET}
                        Change python log level (default: INFO)
  --gen-reference-config
                        Generate reference config from system. Writes to ./reference_config.json. (default: False)
  --skip-sudo           Skip plugins that require sudo permissions (default: False)

Execution Methods

Node Scraper can operate in two modes: LOCAL and REMOTE, determined by the --sys-location argument.

• LOCAL (default): Node Scraper is installed and run directly on the target system. All data collection and plugin execution occur locally.
• REMOTE: Node Scraper runs on your local machine but targets a remote system over SSH. In this mode, Node Scraper does not need to be installed on the remote system; all commands are executed remotely via SSH.

To use remote execution, specify --sys-location REMOTE and provide a connection configuration file with --connection-config.

Example: Remote Execution

node-scraper --sys-name <remote_host> --sys-location REMOTE --connection-config ./connection_config.json run-plugins DmesgPlugin

Example: connection_config.json

{
    "InBandConnectionManager": {
        "hostname": "remote_host.example.com",
        "port": 22,
        "username": "myuser",
        "password": "mypassword",
        "key_filename": "/path/to/private/key"
    }
}

Notes:

• If using SSH keys, specify key_filename instead of password.
• The remote user must have permissions to run the requested plugins and access required files. If needed, use the --skip-sudo argument to skip plugins requiring sudo.

Subcommands

Plugins to run can be specified in two ways, using a plugin JSON config file or using the 'run-plugins' sub command. These two options are not mutually exclusive and can be used together.

'describe' subcommand

You can use the describe subcommand to display details about built-in configs or plugins. List all built-in configs:

node-scraper describe config

Show details for a specific built-in config

node-scraper describe config <config-name>

List all available plugins**

node-scraper describe plugin

Show details for a specific plugin

node-scraper describe plugin <plugin-name>

'run-plugins' sub command

The plugins to run and their associated arguments can also be specified directly on the CLI using the 'run-plugins' sub-command. Using this sub-command you can specify a plugin name followed by the arguments for that particular plugin. Multiple plugins can be specified at once.

You can view the available arguments for a particular plugin by running node-scraper run-plugins <plugin-name> -h:

usage: node-scraper run-plugins BiosPlugin [-h] [--collection {True,False}] [--analysis {True,False}] [--system-interaction-level STRING]
                                            [--data STRING] [--exp-bios-version [STRING ...]] [--regex-match {True,False}]

options:
  -h, --help            show this help message and exit
  --collection {True,False}
  --analysis {True,False}
  --system-interaction-level STRING
  --data STRING
  --exp-bios-version [STRING ...]
  --regex-match {True,False}

Examples

Run a single plugin

node-scraper run-plugins BiosPlugin --exp-bios-version TestBios123

Run multiple plugins

node-scraper run-plugins BiosPlugin --exp-bios-version TestBios123 RocmPlugin --exp-rocm TestRocm123

Run plugins without specifying args (plugin defaults will be used)

node-scraper run-plugins BiosPlugin RocmPlugin

Use plugin configs and 'run-plugins'

node-scraper run-plugins BiosPlugin

'gen-plugin-config' sub command

The 'gen-plugin-config' sub command can be used to generate a plugin config JSON file for a plugin or list of plugins that can then be customized. Plugin arguments which have default values will be prepopulated in the JSON file, arguments without default values will have a value of 'null'.

Examples

Generate a config for the DmesgPlugin:

node-scraper gen-plugin-config --plugins DmesgPlugin

This would produce the following config:

{
  "global_args": {},
  "plugins": {
    "DmesgPlugin": {
      "collection": true,
      "analysis": true,
      "system_interaction_level": "INTERACTIVE",
      "data": null,
      "analysis_args": {
        "analysis_range_start": null,
        "analysis_range_end": null,
        "check_unknown_dmesg_errors": true,
        "exclude_category": null,
        "interval_to_collapse_event": 60,
        "num_timestamps": 3
      }
    }
  },
  "result_collators": {}
}

Running DmesgPlugin with a dmesg log file:

Instead of collecting dmesg from the system, you can analyze a pre-existing dmesg log file using the --data argument:

node-scraper --run-plugins DmesgPlugin --data /path/to/dmesg.log --collection False

This will skip the collection phase and directly analyze the provided dmesg.log file.

Custom Error Regex Example:

You can extend the built-in error detection with custom regex patterns. Create a config file with custom error patterns:

{
  "global_args": {},
  "plugins": {
    "DmesgPlugin": {
      "analysis_args": {
        "check_unknown_dmesg_errors": false,
        "interval_to_collapse_event": 60,
        "num_timestamps": 3,
        "error_regex": [
          {
            "regex": "MY_CUSTOM_ERROR.*",
            "message": "My Custom Error Detected",
            "event_category": "SW_DRIVER",
            "event_priority": 3
          },
          {
            "regex": "APPLICATION_CRASH: .*",
            "message": "Application Crash",
            "event_category": "SW_DRIVER",
            "event_priority": 4
          }
        ]
      }
    }
  },
  "result_collators": {}
}

Save this to dmesg_custom_config.json and run:

node-scraper --plugin-configs dmesg_custom_config.json run-plugins DmesgPlugin

'compare-runs' subcommand

The compare-runs subcommand compares datamodels from two run log directories (e.g. two nodescraper_log_* folders). By default, all plugins with data in both runs are compared.

Basic usage:

node-scraper compare-runs <path1> <path2>

Exclude specific plugins from the comparison with --skip-plugins:

node-scraper compare-runs path1 path2 --skip-plugins SomePlugin

Compare only certain plugins with --include-plugins:

node-scraper compare-runs path1 path2 --include-plugins DmesgPlugin

Show full diff output (no truncation of the Message column or limit on number of errors) with --dont-truncate:

node-scraper compare-runs path1 path2 --include-plugins DmesgPlugin --dont-truncate

You can pass multiple plugin names to --skip-plugins or --include-plugins.

'summary' sub command

The 'summary' subcommand can be used to combine results from multiple runs of node-scraper to a single summary.csv file. Sample run:

node-scraper summary --search-path /<path_to_node-scraper_logs>

This will generate a new file '/<path_to_node-scraper_logs>/summary.csv' file. This file will contain the results from all 'nodescraper.csv' files from '/<path_to_node-scarper_logs>'.

Configs

A plugin JSON config should follow the structure of the plugin config model defined here. The globals field is a dictionary of global key-value pairs; values in globals will be passed to any plugin that supports the corresponding key. The plugins field should be a dictionary mapping plugin names to sub-dictionaries of plugin arguments. Lastly, the result_collators attribute is used to define result collator classes that will be run on the plugin results. By default, the CLI adds the TableSummary result collator, which prints a summary of each plugin’s results in a tabular format to the console.

{
    "globals_args": {},
    "plugins": {
        "BiosPlugin": {
            "analysis_args": {
                "exp_bios_version": "TestBios123"
            }
        },
        "RocmPlugin": {
            "analysis_args": {
                "exp_rocm_version": "TestRocm123"
            }
        }
    }
}

Global args

Global args can be used to skip sudo plugins or enable/disble either collection or analysis. Below is an example that skips sudo requiring plugins and disables analysis.

  "global_args": {
      "collection_args": {
        "skip_sudo" : 1
      },
      "collection" : 1,
      "analysis" : 0
  },

Plugin config: '--plugin-configs' command

A plugin config can be used to compare the system data against the config specifications:

node-scraper --plugin-configs plugin_config.json

Here is an example of a comprehensive plugin config that specifies analyzer args for each plugin:

{
  "global_args": {},
  "plugins": {
    "BiosPlugin": {
      "analysis_args": {
        "exp_bios_version": "3.5"
      }
    },
    "CmdlinePlugin": {
      "analysis_args": {
        "cmdline": "imgurl=test NODE=nodename selinux=0 serial console=ttyS1,115200 console=tty0",
        "required_cmdline" : "selinux=0"
      }
    },
    "DkmsPlugin": {
      "analysis_args": {
        "dkms_status": "amdgpu/6.11",
        "dkms_version" : "dkms-3.1",
        "regex_match" : true
      }
    },
    "KernelPlugin": {
      "analysis_args": {
        "exp_kernel": "5.11-generic"
      }
    },
    "OsPlugin": {
      "analysis_args": {
        "exp_os": "Ubuntu 22.04.2 LTS"
      }
    },
    "PackagePlugin": {
          "analysis_args": {
            "exp_package_ver": {
              "gcc": "11.4.0"
            },
            "regex_match": false
          }
    },
    "RocmPlugin": {
      "analysis_args": {
        "exp_rocm": "6.5"
      }
    }
  },
  "result_collators": {},
  "name": "plugin_config",
  "desc": "My golden config"
}

Reference config: 'gen-reference-config' command

This command can be used to generate a reference config that is populated with current system configurations. Plugins that use analyzer args (where applicable) will be populated with system data. Sample command:

node-scraper --gen-reference-config run-plugins BiosPlugin OsPlugin

This will generate the following config:

{
  "global_args": {},
  "plugins": {
    "BiosPlugin": {
      "analysis_args": {
        "exp_bios_version": [
          "M17"
        ],
        "regex_match": false
      }
    },
    "OsPlugin": {
      "analysis_args": {
        "exp_os": [
          "8.10"
        ],
        "exact_match": true
      }
    }
  },
  "result_collators": {}

This config can later be used on a different platform for comparison, using the steps at #2:

node-scraper --plugin-configs reference_config.json

An alternate way to generate a reference config is by using log files from a previous run. The example below uses log files from 'scraper_logs_

node-scraper gen-plugin-config --gen-reference-config-from-logs scraper_logs_<path>/ --output-path custom_output_dir

This will generate a reference config that includes plugins with logged results in 'scraper_log_