adqsetup 2.0.2

A pure Python tool and library to setup ADQ for Intel NICs

adqsetup

SPDX-License-Identifier: BSD-3-Clause
Copyright (c) 2022, Intel Corporation

Dependencies: Python 2.7-3.10

Distro Support:

• Redhat: 7.1-9.x
• Fedora: 28-35
• Ubuntu: 19.04-22.04
• Debian: 11

Installation

Python Package Index (pip):

python -m pip install adqsetup

Included with driver:

python scripts/adqsetup/adqsetup.py install

Usage

The basic usage is: adqsetup [options] <command> [parameters ...]
Please see the output of adqsetup help for a complete list of command line options.

Commands

• help: Show help message
• examples: Create an 'examples' subdirectory
  The examples subdirectory - created in the current directory - contains a set of sample config files
• apply {filename}: Apply a config file
  • {filename}: Config file (relative or full path)
    If empty or '-', config file is read from stdin.
• create { [{name}] { {key} {value} }... }...: Create a config from the command line
  Each section consisting of a bracketed name and one or more {key} {value} pairs.
  • [{name}]: User-defined name of section
    Must be unique within a configuration, '[globals]' is reserved but can be used.
  • {key} {value}: Configuration Parameter
    One or more space-seperated key and value pairs. See the above Class Configuration Parameter list for possible keys and values.
• reset: Remove ADQ traffic classes and filters
  Attempts to perform a cleanup of any ADQ-related setup. Note: '--priority=skbedit' option must be included to remove the egress filters.
• persist {filename}: Persist a config file across reboots
  Creates a systemd service unit set to run once on boot after the network is running. One config per network interface, new configs overwrite old ones.
  • {filename}: Config file (relative or full path)
    If empty or '-', config file is read from stdin.
• install: Install the adqsetup script
  Installs the current script at /usr/local/bin

Configuration Parameters

Globals Section

• arpfilter: (bool) Enable selective ARP activity
• bpstop: (bool) Channel-packet-clean-bp-stop feature
• bpstop-cfg: (bool) Channel-packet-clean-bp-stop-cfg feature
• busypoll: (integer) busy_poll value
• busyread: (integer) busy_read value
• cpus: (integer list|'auto') CPUs to use for handling 'default' traffic, default 'auto'
• numa: (integer|'local'|'remote'|'all') Numa node to use for 'default' traffic, default 'all' (prefer local)
• dev: (string) Network interface device to configure
• optimize: (bool) Channel-inspect-optimize feature
• priority: ('skbedit') Method to use for setting socket priority, default none
• queues: (integer) Number of queues in 'default' traffic class, default 2
• txring: (integer) Transmit ring buffer size
• txadapt: (bool) Adaptive transmit interrupt coalescing
• txusecs: (integer) Usecs for transmit interrupt coalescing
• rxring: (integer) Receive ring buffer size
• rxadapt: (bool) Adaptive receive interrupt coalescing
• rxusecs: (integer) Usecs for receive interrupt coalescing

User-defined Section (for each application or traffic class)

• addrs: (string list) Local IP addresses of traffic
• cpus: (integer list|'auto') CPUs to use for handling traffic, default 'auto'
• mode: ('exclusive'|'shared') Mode for traffic class
• numa: (integer|'local'|'remote'|'all') Numa node to use for traffic, default 'all' (prefer local)
• pollers: (integer) Number of independent pollers, default 0
• poller-timeout: (integer) Independent poller timeout value, default 10000
• ports: (integer list) Local IP ports of traffic
• protocol: ('tcp'|'udp') IP Protocol of traffic
• queues: (integer) Number of queues in traffic class
• remote-addrs: (string list) Remote IP addresses of traffic
• remote-ports: (integer list) Remote IP ports of traffic

Sample Usage

adqsetup help
adqsetup examples

adqsetup apply memcached.conf
adqsetup --dev=eth4 apply nginx.conf
adqsetup --dev=eth3 persist eth3.conf

cat memcached.conf | adqsetup apply

adqsetup create [myapp] queues 4 ports 11211

adqsetup --verbose create \
  [globals] priority skbedit \
  [myapp] queues 2 ports 11211

adqsetup --verbose create \
  [app1] mode shared queues 4 ports 6379-6382
  [app2] queues 2 ports 11211 pollers 2

Sample Usage Bash Script

#!/bin/bash 
QUEUES=8
# this will loop through a range
# of busy_poll values
for BP in {10000..50000..5000}; do 
    adqsetup create [globals] busypoll $BP [nginx] queues $QUEUES ports 80
    # run test here
done

Sample Usage With Pipes From Bash Script

#!/bin/bash 
QUEUES=8
# this will loop through a range
# of busy_poll values
for BP in {10000..20000..5000}; do 
adqsetup apply <<EOF
    [globals]
    dev=eth2
    busypoll=$BP
    [nginx]
    queues=$QUEUES
    ports=80
EOF
# run test here
done

Sample Usage With Pipes From External Script

python makeconf.py | adqsetup --json apply

makeconf.py

import json
conf = {
    "globals": {
        "dev": "eth2",
        "busypull": 10000
    },
    "app1": {
        "queues": 4,
        "ports": "80,443"
    }
}
print(json.dumps(conf))

Notes

• To load/use a different device driver while creating the setup, the --driver parameter may be used. Device driver path is the full path to the .ko file (ex: ice-1.9.x/src/ice.ko). Interface must be set to come up automatically with an ip address (via NetworkManager or other). adqsetup will wait up to three seconds for this to occur before erroring out. Conversely, you can load the driver and setup the interface manually before running the adqsetup.

• The independent pollers argument passed to adqsetup doesn’t map directly to the qps_per_poller arguments passed to the driver. adqsetup allows the user to specify how many pollers for a particular TC instead of having to specify qps_per_poller.

• adqsetup 1.x required updated versions of the 'tc', 'ethtool', and 'devlink' commands to be installed on the system. With adqsetup 2.x and onward, this requirement has been removed.

Common Issues

• If you get a /usr/bin/env: ‘python’: No such file or directory error when you run the script, please install Python. If you have already installed Python, then try whereis python and you should see a message like: python: /usr/bin/python2.7 /usr/bin/python3.6 /usr/bin/python3.6m /usr/bin/python3.9 on the first line of the output. Either run the version you wish to use manually: python3.6 adqsetup.py help, or create a 'python' symbolic link on the path: ln -s /usr/bin/python3.6 /usr/local/bin/python

• Many advanced features, such as pollers and the per-tc flow director may not be supported by older versions of the driver or kernel. adqsetup will attempt to use an equivalent fallback feature, and if none are available a descriptive error will be provided. Please refer to the ADQ Config Guide for more information.

Other Issues

Please run the malfunctioning config with the command line --debug option, which should include a short stack trace at the end of the output. Send the configuration file (if used), full commmand line, and program output to your Intel support contact.

JSON Support

adqsetup accepts configurations in the JSON format from either a file or stdin with the --json option. Parameters are the same as listed above, using the following basic structure:

{
    "globals": {
        "dev": "eth4",
        "priority": "skbedit"
    },
    "app1": {
        "queues": 2,
        "ports": 11211
    },
    "app2": {
        "queues": 4,
        "mode": "shared",
        "ports": "6379-6382"
    }
}