pytest-bluezenv 0.1.4

pytest BlueZ environment plugin

pytest-bluezenv Pytest plugin is used for functional testing of BlueZ and kernel using multiple virtual machine environments, connected by real or virtual controllers.

OPTIONS

The pytest-bluezenv plugin has command-line options:

–kernel=<image>:

Kernel image (or built Linux source tree root) to use. See test-runner(1) and tester.config for required kernel config.

If not provided, value from FUNCTIONAL_TESTING_KERNEL environment variable is used. If none, no image is used.

–usb=hci0,hci1:

USB controllers to use in tests that require use of real controllers.

If not provided, value from FUNCTIONAL_TESTING_CONTROLLERS environment variable is used. If none, all USB controllers with suitable permissions are considered.

–btmon:

Launch btmon on all hosts to log events, and dump traffic to test-bluezenv-*.btsnoop

–force-usb:

Force tests to use USB controllers instead of btvirt.

–vm-timeout=<seconds>:

Specify timeout for communication with VM hosts.

–log-filter=[+-]<pattern>,[+-]<pattern>,…:

Allow/deny lists for filtering logging output. The pattern is a shell glob matching to the logger names.

–build-dir=<path>:

Path to build directory where to search for BlueZ executables.

–kernel-build=no/use/auto/force:

Build a suitable kernel image from source.

–kernel-upstream=<GIT_URL>:

URL for Git clone of kernel sources.

–kernel-branch=<GIT_BRANCH>:

Git branch to build from.

Tests that require kernel image or USB controllers are skipped if none are available. Normally, tests use btvirt.

VM instances share a directory /run/shared with host machine, located on host usually in /tmp/pytest-bluezenv-*/shared-*. Core dumps etc. are copied out from it before test instance is shut down.

REQUIREMENTS

General

The following are needed:

• QEmu (x86_64)

• dbus-daemon available

Recommended:

• KVM-enabled x86_64 host system

• Preferably built BlueZ source tree

• chronyd available

• util-linux tools available

• agetty available

Kernel

Running VM-based tests requires a kernel image with similar config as BlueZ test-runner(1). If given –kernel-build option, a suitable image is built from sources downloaded under .pytest_cache.

Simplest setup is

cp ../bluez/doc/tester.config .config
make olddefconfig
make -j8

To get log timestamps right, the kernel should have the following configuration enabled:

CONFIG_HYPERVISOR_GUEST=y
CONFIG_PARAVIRT=y
CONFIG_KVM_GUEST=y

CONFIG_PTP_1588_CLOCK=y
CONFIG_PTP_1588_CLOCK_KVM=y
CONFIG_PTP_1588_CLOCK_VMCLOCK=y

USB

Some tests may require a hardware controller instead of the virtual btvirt one.

EXAMPLES

Run all tests

$ python3 -mpytest --kernel=/pathto/bzImage

$ export FUNCTIONAL_TESTING_KERNEL=/pathto/bzImage
$ python3 -mpytest

Show output during run

$ python3 -mpytest --log-cli-level=0

Show only specific loggers:

$ python3 -mpytest --log-cli-level=0 --log-filter=rpc,host

$ python3 -mpytest --log-cli-level=0 --log-filter=*.bluetoothctl

Filter out loggers:

$ python3 -mpytest --log-cli-level=0 --log-filter=-host

$ python3 -mpytest --log-cli-level=0 --log-filter=host,-host.*.1

Run selected tests

$ python3 -mpytest test/functional/test_cli_simple.py::test_bluetoothctl_script_show

$ python3 -mpytest -k test_bluetoothctl_script_show

$ python3 -mpytest -k 'test_btmgmt or test_bluetoothctl'

Don’t run tests with a given marker:

$ python3 -mpytest -m "not pipewire"

Don’t run known-failing tests:

$ python3 -mpytest -m "not xfail"

Note that otherwise known-failing tests would be run, but with failures suppressed.

Run previously failed and stop on failure

$ python3 -mpytest -x --ff

Show errors from know-failing test

$ python3 -mpytest --runxfail -k test_btmgmt_info

Redirect USB devices

$ python3 -mpytest --usb=hci0,hci1

$ export FUNCTIONAL_TESTING_CONTROLLERS=hci0,hci1
$ python3 -mpytest -vv

This does not require running as root. Changing device permissions is sufficient. In verbose mode (-vv) some instructions are printed.

Run all tests using the USB controllers:

$ python3 -mpytest --usb=hci0,hci1 --force-usb

Run tests in parallel

pytest-xdist is required for parallel execution. To run:

$ python3 -mpytest -n auto

To reduce VM setup/teardowns:

$ python3 -mpytest -n auto --dist loadgroup

Logging in to a test VM instance

While test is running:

$ python3 -mpytest_bluezenv attach

For this to be useful, usually, you need to pause the test e.g. by running with --trace option.

To do it manually, when starting the tester will log a line like:

TTY: socat /tmp/pytest-bluezenv-q658swgi/pytest-bluezenv-tty-0 STDIO,rawer

with the location of the socket where the serial is connected to.

WRITING TESTS

The functional tests are written in files (test modules) names test/functional/test_*.py. They are written using standard Pytest style. See https://docs.pytest.org/en/stable/getting-started.html

Use Black to autoformat Python test code.

Example: Virtual machines

from pytest_bluez import host_config, Bluetoothd, Bluetoothctl

@host_config(
    [Bluetoothd(), Bluetoothctl()],
    [Bluetoothd(), Bluetoothctl()],
)
def test_bluetoothctl_pair(hosts):
    host0, host1 = hosts

    host0.bluetoothctl.send("scan on\n")
    host0.bluetoothctl.expect(f"Controller {host0.bdaddr.upper()} Discovering: yes")

    host1.bluetoothctl.send("pairable on\n")
    host1.bluetoothctl.expect("Changing pairable on succeeded")
    host1.bluetoothctl.send("discoverable on\n")
    host1.bluetoothctl.expect(f"Controller {host1.bdaddr.upper()} Discoverable: yes")

    host0.bluetoothctl.expect(f"Device {host1.bdaddr.upper()}")
    host0.bluetoothctl.send(f"pair {host1.bdaddr}\n")

    idx, m = host0.bluetoothctl.expect(r"Confirm passkey (\d+).*:")
    key = m[0].decode("utf-8")

    host1.bluetoothctl.expect(f"Confirm passkey {key}")

    host0.bluetoothctl.send("yes\n")
    host1.bluetoothctl.send("yes\n")

    host0.bluetoothctl.expect("Pairing successful")

The test declares a VM setup with two Qemu instances, where both hosts run bluetoothd and start a bluetoothctl process. The Qemu instances have btvirt virtual BT controllers and can see each other.

The test itself runs on the parent host.

The host0/1.bluetoothctl.* commands invoke RPC calls to one of the the two VM instances. In this case, they are controlling the bluetoothctl process using pexpect library to deal with its command line.

When the test body finishes executing, the test passes. Or, it fails if any assert statement fails or an error is raised. For example, above RemoteError due to bluetoothctl not proceeding as expected in pairing is possible.

The host configuration (bluetoothd + bluetoothctl above) is torn down between test (SIGTERM/SIGKILL sent etc.).

By default the VM instance itself continues running, and may be used for other tests that share the same VM setup.

Generally, the framework automatically orders the tests so that the VM setup does not need to be restarted unless needed.

Example host plugin

The host.bluetoothctl implementation used above is as follows:

from pytest_bluez import HostPlugin, Bluetoothd

class Bluetoothctl(Pexpect):
    # Declare unique plugin name
    name = "bluetoothctl"

    # Declare plugin dependencies to be loaded first
    depends = [Bluetoothd()]

    # These run on parent host side:

    def __init__(self, subdir, name):
        self.exe = utils.find_exe(subdir, name)

    def presetup(self):
        pass

    # These run on VM side at setup/teardown:

    def setup(self, impl):
        self.log = logging.getLogger(self.name)
        self.log_stream = utils.LogStream(self.name)
        self.ctl = pexpect.spawn(self.exe, logfile=self.log_stream.stream)

    def teardown(self):
        self.ctl.terminate()

    # These define custom RPC methods that can be called

    def expect(self, *a, **kw):
        ret = self.ctl.expect(*a, **kw)
        self.log.debug("match found")
        return ret, self.ctl.match.groups()

    def send(self, *a, **kw):
        return self.ctl.send(*a, **kw)

Host plugins are for injecting code to run on the VM side test hosts. The host plugins have scope of one test. The VM side test framework sends SIGTERM and SIGKILL to all processes in the test process group to reset the state between each test.

The plugins are declared by inheriting from HostPlugin. Their __init__() is supposed to only store declarative configuration on self and runs on parent side early in the test discovery phase. The presetup runs on parent side in test setup phase, before VM environment is started. The plugin can for example do pytest.skip(reason=”something”) to skip the test.

The setup() and teardown() methods run on VM-side at host environment start and end. All other methods can be invoked via RPC by the parent tester, and any values returned by them are passed via RPC back to the parent.

To load a plugin to a VM host, pass it to host_config() in the declaration of a given test.

Test fixtures

The following test fixtures are used to deal with spawning VM hosts:

hosts

Session-scope fixture that expands to a list of VM host proxies
(`HostProxy`), with configuration as specified in `host_config`. The
VM instances used may be reused by other tests.  The userspace test
runner is torn down between tests.

Example:

    def test_something(hosts):
        host0 = hosts[0]
        host1 = hosts[1]

hosts_once

def test_something(hosts_once):
    host0 = hosts_once[0]
    host1 = hosts_once[1]

Function-scope fixture. Same as hosts, but spawn separate VM instances for this test only.

Others

The following fixtures are defined, but mainly for use as dependencies to hosts: kernel (selected kernel image), usb_indices (selected USB controllers), host_setup (current host plugin configurations), vm_setup (VM host configuration), vm (VM instances without userspace setup), vm_once (same but with function scope).

Utilities

In addition to standard Pytest features, the following items are available in the pytest_bluez module.

host_config

@host_config(*host_setup, hw=False, reuse=False)

Declare host configuration.

Args:
    *host_setup: each argument is a list of plugins to be loaded on a host.
        The number of arguments specifies the number of hosts.
    hw (bool): whether to require hardware BT controller
    reuse (bool): whether to define a setup where the test host processes
        are not required to be torn down between tests. This is only useful
        for tests that do not perturb e.g. bluetoothd state too much.

Returns:
    callable: decorator setting pytest attributes

Example:

    @host_config([Bluetoothd()], [Bluetoothd()])
    def test_something(hosts):
        host0, host1 = hosts

Example:

    # Allow not restarting Bluetoothd between tests sharing this configuration
    base_config = host_config([Bluetoothd()], reuse=True)

    @base_config
    def test_one(hosts):
        host0, = hosts

    @base_config
    def test_two(hosts):
        # Note: uses same Bluetoothd() instance as above
        host0, = hosts

parametrized_host_config

Declare parametrized host configurations.

See https://docs.pytest.org/en/stable/how-to/parametrize.html for the
concept.

Args:
    param_host_setups (list): list of host setups
    hw (bool): whether to require hardware BT controller
    reuse (bool): whether to define a setup where the test host processes
        are not required to be torn down between tests. This is only useful
        for tests that do not perturb e.g. bluetoothd state too much.

Returns:
    callable: decorator setting pytest attributes

HostProxy

class HostProxy:
    """
    Parent-side proxy for VM host: load plugins, RPC calls to plugins
    """

    def load(self, plugin: HostPlugin):
        """
        Load given plugin to the VM host synchronously.
        """

    def start_load(self, plugin: HostPlugin):
        """
        Initiate loading the given plugin to the VM host.  Use
        `wait_load` to wait for completion and make loaded plugins
        usable.

        """

    def wait_load(self):
        """
        Wait for plugin loads to complete, and make plugins available.
        """

    def close(self)
        """
        Shutdown this VM host tester instance.
        """

    def __getattr__(self, name):
        """
     Get a proxy attribute for one of the loaded plugins
     """

Parent host-side representation of one VM host with loadable plugins.

Plugins are usually loaded based on host_setup, but can also be loaded during the test itself.

Loaded plugins appear as attributes on the host proxy.

find_exe

from pytest_bluez import find_exe
bluetoothctl = find_exe("client", "bluetoothctl")

Find absolute path to the given executable, either within BlueZ build directory or on host.

mainloop_invoke

Blocking invoke of `func` in GLib main loop.

Note:

    GLib main loop is only available for VM host plugins, not in tester.

Example:

    value = mainloop_invoke(lambda: 123)
    assert value == 123

Warning:
    dbus-python **MUST** be used only from the GLib main loop,
    as the library has concurrency bugs. All functions using it
    **MUST** either run from GLib main loop eg. via mainloop_wrap

mainloop_wrap

Wrap function to run in GLib main loop thread

Note:

    GLib main loop is only available for VM host plugins, not in tester.

Example:

    @mainloop_wrap
    def func():
        bus = dbus.SystemBus()

mainloop_wrap

Wrap function to assert it runs from GLib main loop

Note:

    GLib main loop is only available for VM host plugins, not in tester.

Example:

    @mainloop_assert
    def func():
        bus = dbus.SystemBus()

LogStream

from pytest_bluez import LogStream

log_stream = LogStream("bluetoothctl")
subprocess.run(["bluetoothctl", "show"], stdout=log_stream.stream)

Utility to redirect a stream to logging with accurate kernel-provided timestamps.

RemoteError

from pytest_bluez import RemoteError

try:
    host.call(foo)
except RemoteError as exc:
    print(exc.traceback)
    original_exception = exc.exc

Exception raised on the VM side, passed through RPC. Properties: traceback is a traceback string and exc is the original exception instance raised on the remote side.

Host plugins

The following host plugins are available:

HostPlugin

Base class for host plugins. See also example above.

class HostPlugin:
    """
    Plugin to insert code to VM host side.

    Attributes:
        name (str): unique name for the plugin
        depends (tuple[HostPlugin]): plugins to be loaded before this one
        value (object): object to appear as HostProxy attribute on parent side.
            If None, the plugin is represented by a proxy object that does RPC
            calls. Otherwise, must be a serializable value.

    """

    name = None
    depends = ()
    value = None

    def __init__(self):
        """
        Configure plugin (runs on parent host side).  This is
        called at test discovery time, so should mainly store static
        data.

        """
        pass

    def presetup(self):
        """
        Parent host-side setup, before VM environment is started.  May
        use pytest.skip() to skip tests in case plugin cannot be set up.

        """
        pass

    def setup(self, impl):
        """
        VM-side setup

        Args:
            impl (Implementation): plugin host object
        """
        pass

    def teardown(self):
        """VM-side teardown"""
        pass

Agent

DBus org.bluez.Agent1 test implementation.

class Agent(env.HostPlugin):
    """
    Host plugin providing org.bluez.Agent1 test implementation.

    Asynchronous events are handled via expect().

    Example:

        host.agent.device_method(host1.bdaddr, "Pair")
        event = host.agent.expect("org.bluez.Agent1.RequestConfirmation")
        assert event.passkey == 1234
        host.agent.reply()
    """

    depends = [Bluetoothd()]
    name = "agent"

    def __init__(self, capability="KeyboardDisplay", path="/agent"):

    def has_device(self, address):
        """
        Return True if device with given address exists
        """

    def device_method(self, address, method, *a, **kw):
        """
        Call given org.bluez.Device1 DBus method

        Args:
            address (str): bdaddr of target device
            method (str): name of DBus method, without interface prefix
            *a, **kw: argument passed to the DBus method call

        Events:
            AgentEvent(kind="org.bluez.Device1.{method}:reply")
        """

    def adapter_method(self, method, *a, **kw):
        """
        Call given org.bluez.Adapter1 DBus method

        Args:
            method (str): name of DBus method, without interface prefix
            *a, **kw: argument passed to the DBus method call

        Events:
            AgentEvent(kind="org.bluez.Adapter1.{method}")
        """

    def adapter_set(self, key, value):
        """
        Set given org.bluez.Adapter1 property
        """

    def adapter_get(self, key):
        """
        Get given org.bluez.Adapter1 property
        """

    def get_event(self, block=True):
        """
        Get most recent pending AgentEvent, blocking optional
        """

    def expect(self, kinds):
        """
        Get most recent pending AgentEvent and assert its kind

        Returns:
            event (AgentEvent)
        """

    def reply(self, *value):
        """
        Provide DBus reply to the most recent pending AgentEvent

        Arguments:
            *value: DBus reply return values
        """

    def reply_error(self, err=None):
        """
        Provide DBus error reply to the most recent pending AgentEvent

        Arguments:
            err (dbus.DBusException): DBus error. Default: org.bluez.Error.Rejected
        """

class Event:
    """
    Asynchronous event

    Properties:
        kind (str): event kind
        info (dict): event properties (also available as attributes)
    """

class EventPluginMixin:
    """
    Simple expect() / reply() pattern for handing async events in
    host plugins.
    """

def dbus_service_event_method(
    interface, name, args=(), in_signature="", out_signature="", sync=True
):
    """
    dbus.service.method that pushes Event instances to self.events

    Example:

        class AgentObject(dbus.service.Object):
            @utils.mainloop_assert
            def __init__(self, bus, path, events):
                self.events = events
                super().__init__(bus, path)

            AuthorizeService = dbus_service_event_method(
                "org.bluez.Agent1",
                "AuthorizeService", ("device", "uuid"), "os", sync=False
            )

    """

Bdaddr

Host plugin providing host.bdaddr. Loaded by default.

Bluetoothctl

class Bluetoothctl(HostPlugin)
    def expect(self, *a, **kw)
    def send(self, *a, **kw)

Host plugin for starting and controlling bluetoothctl with pexpect.

Bluetoothd

Host plugin starting Bluetoothd.

Btmon

Host plugin providing btmon running in the background. Usually should be loaded via –btmon.

Call

class Call(HostPlugin)

Host plugin providing ``host.call(func, *args, **kw)`` and `call_async`
which invoke the given functions on VM host side.  Loaded by default.

Example:

    result = host0.call(my_func, 1, 2, 3)

Example:

    result_async = host0.call(my_func, 1, 2, 3, sync=False)
    ...
    result = result_async.wait()

DbusSession

Host plugin providing session DBus, at address impl[“dbus-session”].address.

DbusSystem

Host plugin providing system DBus, at address impl[“dbus-system”].address.

Pexpect

class Pexpect(env.HostPlugin)

Host plugin for starting and controlling processes with pexpect.

Example:

    btmgmt = host0.pexpect.spawn(find_exe("tools", "btmgmt"))
    btmgmt.send("info\n")
    btmgmt.expect("hci0")
    btmgmt.close()

Rcvbuf

Host plugin setting pipe buffer size defaults. Loaded by default.