byteblower-test-framework 1.1.2

Test Framework for the ByteBlower Traffic Generator.

An easy accessible library of basic test flows and reporting engines.

ByteBlower ® is a traffic generator/analyzer system for TCP/IP networks.

This library provides you with the building blocks for:

1. Generating test traffic

2. Collect statistics

3. Analyze traffic statistics

4. Generate reports

Release notes

What can this version bring to you? See our exciting new and existing features below!

✨ Test Cases ✨

The ByteBlower Test Framework serves as the base for many of our customer’s test cases.

At Excentis, we also develop common test cases and love to share them with the community.

• Test Case: RFC 2544 Throughput

  • Run an RFC 2544 network performance test with ease!

• Test Case: Low Latency

  • Run low latency validation tests on your network.

📢 New since v1.1.0! 📢

It is with great pleasure that we announce our new features of the ByteBlower Test Framework!

• Command-line interface

  • Run traffic tests with nothing more than a JSON configuration file!

• 🚧 Feature previews 🚧

  • The HTML and JSON report include flow runtime error information.

  • Most building blocks now have an option to explicitly release related resources on the ByteBlower system.

Features

• Quick and easy automation of traffic tests using ByteBlower

• Straightforward building blocks for your own test scripts

  • Grouped in self-explaining categories

  • Designed with a small 😉 to the workflow you are used to from the GUI

• Supported ByteBlower endpoint types

  • ByteBlower Port (on a physical interface of a ByteBlower server)

• Endpoint configuration

  • IPv4: DHCP and manual address configuration

    • Automatic handling of endpoints located behind a NA(P)T gateway!

  • IPv6: DHCPv6 and SLAAC address configuration

• Traffic types

  • Stateless: frame blasting (UDP-based)

  • Stateful: HTTP (TCP-based)

• Application simulations

  • Voice over IP (VoIP) using the G.711 codec

  • Video streaming (Netflix, YouTube, …)

• Standard analysis

  • Frame count: transmitted and received over time, frame loss and byte loss

  • Latency: Minimum, maximum, average and jitter over time

  • Aggregation of results over multiple flows (frame blasting only in this release)

  • PASS/FAIL criteria can be provided to match your KPIs

• Application-specific analysis

  • Mean Opinion Score (MOS): Specific for Voice flows

  • Video buffer analysis: Specific for Video flows

• Reporting

  • Summary and realtime results

  • HTML: Neat to share with your chief, customers, vendors, …

    • Incorporating our brand new style!

    • Interactive charts

    • Includes overview of all latency CCDF results

  • JSON: Allows for machine post-processing

  • XML JUnit: Useful for integration in automation tools

• Helpers

  • Ease-of-use for configuration and/or post-processing of the analyzed results

• Example scripts

Changelog

For all details, please have a look at our Changelog.

Requirements

• byteblowerll (ByteBlower API): Our lower layer API for client-server communication (API documentation)

• scapy: Used for frame generation and parsing

• junit-xml: Used for Unit test report generation

• pandas: Used for data collection

• highcharts-excentis: Used for generating graphs

• jinja2: User for HTML report templating

Supported platforms

The ByteBlower Test Framework in general supports Python version 3.7 to 3.11.

Note

NOTE: Python >= 3.12 is not yet supported because the ByteBlower API libraries are not yet available for Python 3.12 (byteblowerll).

The framework has been tested for the following operating system platforms and Python versions:

OS platform	Distribution	Python version	source
Windows 10	up to feature release 21H2	Python 3.10	Official Python
Windows 10	up to feature release 21H2	Python 3.9	Official Python
Windows 10	up to feature release 21H2	Python 3.8	Official Python
Windows 10	up to feature release 21H2	Python 3.7	Official Python
Windows 10	up to feature release 21H2	Python 3.9	Windows Apps
Windows 10	up to feature release 21H2	Python 3.8	Windows Apps
Windows 10	up to feature release 21H2	Python 3.7	Windows Apps
macOS	up to Monterey	Python 3.9	Official Python (Intel-only!)
macOS	up to Monterey	Python 3.8	Official Python (Intel-only!)
Linux	Debian 11 (bullseye)	Python 3.9.2	Debian packages
Linux	Debian 10 (buster)	Python 3.7.3	Debian packages
Linux	Ubuntu 20.04 (Focal Fossa)	Python 3.8.2	Ubuntu packages
Linux	Ubuntu 22.04 (Focal Fossa)	Python 3.10.4	Ubuntu packages
Docker	python:3.10-slim-buster	Python 3.10.11	Docker Python
Docker	python:3.9-slim-buster	Python 3.9.16	Docker Python
Docker	python:3.8-slim-buster	Python 3.8.16	Docker Python
Docker	python:3.7-slim-buster	Python 3.7.13	Docker Python

Installation

Prepare runtime environment

We recommend managing the runtime environment in a Python virtual environment. This guarantees proper separation of the system-wide installed Python and pip packages.

Python virtual environment

Make sure to use the right Python version (>= 3.7, <= 3.11), list all Python versions installed in your machine by running:

1. On Windows systems using PowerShell:

   py --list

If no Python version is in the required range, you can download and install Python 3.7 or above using your system package manager or from https://www.python.org/ftp/python.

Prepare Python virtual environment: Create the virtual environment and install/update pip and build.

1. On Unix-based systems (Linux, WSL, macOS):

   Note: Mind the leading . which means sourcing ./env/bin/activate.

   python3 -m venv --clear env
   . ./env/bin/activate
   pip install -U pip build

2. On Windows systems using PowerShell:

   Note: On Microsoft Windows, it may be required to enable the Activate.ps1 script by setting the execution policy for the user. You can do this by issuing the following PowerShell command:

   PS C:> Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

   See About Execution Policies for more information.

   py -3.8 -m venv --clear env
   & ".\env\Scripts\activate.ps1"
   python -m pip install -U pip build

Install the ByteBlower Test Framework from PyPI

First make sure that your activated your virtual environment:

1. On Unix-based systems (Linux, WSL, macOS):

   . ./env/bin/activate

2. On Windows systems using PowerShell:

   & ".\env\Scripts\activate.ps1"

Now install (or update) the ByteBlower Test Framework:

pip install -U byteblower-test-framework

Documentation

Online usage documentation: ByteBlower Test Framework documentation

The API documentation is also always available in the API:

help(any_api_object)

Some examples:

For classes (and their members):

from byteblower_test_framework.host import Server
from byteblower_test_framework.endpoint import IPv4Port
from byteblower_test_framework.traffic import FrameBlastingFlow

help(Server)
help(Server.start)
help(Server.info)
help(IPv4Port)
help(FrameBlastingFlow)

from byteblower_test_framework.report import ByteBlowerHtmlReport

help(ByteBlowerHtmlReport)

For objects (and their members):

from byteblower_test_framework.host import Server

my_server = Server('byteblower-39.lab.excentis.com.')

help(my_server)
help(my_server.start)

Usage

First make sure that your activated your virtual environment:

1. On Unix-based systems (Linux, WSL, macOS):

   . ./env/bin/activate

2. On Windows systems using PowerShell:

   & ".\env\Scripts\activate.ps1"

Let’s give it a test run: Import the test framework and show its documentation:

python

import byteblower_test_framework
help(byteblower_test_framework)

This shows you the ByteBlower Test Framework module documentation.

Command-line interface

To get help for command line arguments:

1. As a python module:

   python -m byteblower_test_framework --help

2. As a command-line script:

   byteblower-test-framework --help

For a quick start, you can run a simple test using the JSON configuration provided below:

{
    "server": "byteblower.example.com.",
    "ports": [
        {
            "name": "CMTS",
            "port_group": [
                "source_group"
            ],
            "interface": "nontrunk-1",
            "ipv4": "dhcp"
        },
        {
            "name": "CM-LAN",
            "port_group": [
                "destination_group"
            ],
            "interface": "trunk-1-1",
            "ipv4": "dhcp",
            "nat": true
        }
    ],
    "flows": [
        {
            "name": "Downstream UDP",
            "source": {
                "port_group": [
                    "source_group"
                ]
            },
            "destination": {
                "port_group": [
                    "destination_group"
                ]
            },
            "type": "frame_blasting",
            "frame_size": 60,
            "frame_rate": 8500,
            "add_reverse_direction": true,
            "analysis": {
                "latency": true
            }
        },
        {
            "name": "Upstream HTTP",
            "source": {
                "port_group": [
                    "destination_group"
                ]
            },
            "destination": {
                "port_group": [
                    "source_group"
                ]
            },
            "add_reverse_direction": true,
            "type": "http"
        }
    ],
    "report": {
        "html": true,
        "json": false,
        "junit_xml": false
    },
    "maximum_run_time": 10.0
}

Please make sure you change the server and ports configuration according to the setup you want to run your test on.

Copy this configuration into byteblower_test_framework.json. Then, run the test in the command line interface using:

byteblower-test-framework

The resulting reports will be saved into the current directory. You can specify a different config file name and report path using:

byteblower-test-framework --config-file path/to/my_test_config.json  --report-path path/to/my_test_reports_directory

You can find more details on how to customize your own configuration file in Configuration file.

Note

To-do: We will provide a quick start guide in the future.

Development

Would you like to contribute to this project? You’re very welcome! 😊

Please contact us at ByteBlower Support and we’ll be there to guide you.

Support

If you have any questions or feature request you can contact the ByteBlower support team using:

🌍: Excentis Support Portal

🖂: ByteBlower Support

🕽: [phone]