wirepas-backend-client 1.3.5rc3

Interfaces to interact with Wirepas backend services.

Backend-client

1. Introduction
2. Installation
   1. Host dependencies
   2. Setting up a Python virtual environment
   3. Installing from PyPi
   4. Installing from Github
3. Entrypoints
   1. Gateway command line interface
   2. WPE Viewer
   3. WNT Viewer
   4. Provisioning server
4. Framework
   1. Structure
   2. Examples
5. Logging to fluentd
6. Building and running over Docker
   1. Dockerhub
   2. Running with docker
   3. Running with compose
   4. Building the image locally
7. Source documentation
8. Contributing
9. License

Introduction

The Backend Client is a tool to serve as a decoder and a test framework where you can extend the existing interfaces to develop your own test cases. Interactions with a gateway must be compliant with the Wirepas Backend to Gateway API.

In addition to the interaction with the Backend to Gateway Public API, the Backend Client has also support for Wirepas Positioning Engine and Wirepas Network Tool. Support for these tools is still work in progress (see milestones and projects). However, you can already benefit from existing entrypoints to help you capture data and debug your integration (see entrypoints).

Figure 1 illustrates the horizon where Backend Client is located.

Figure 1 - Backend Client role in the Wirepas ecosystem.

Installation

The Backend Client is written in python and it requires a valid MQTT broker target to connect to, WNT or WPE credentials.

If you need help installing a MQTT broker or auxiliary software please refer to our tutorials repository.

Host dependencies

The main requirements of Backend Client are:

• Python 3.7
• Pip (we recommend the latest available)
• Linux system

Certain Backend Client's dependencies must be compiled locally upon installation, thus require the installation of specific development headers.

Under Debian, the extra requirements are met with:

    sudo apt-get install default-libmysqlclient-dev \
                         gcc \
                         python3-dev

Please be advised that the name of such packages might change depending on your Linux system. Please refer to the Dockerfiles under the container folder for more detailed information.

Windows native environments are not supported. For help running the tool in Windows, please use the Linux subsystem, Docker or a virtual machine.

Setting up a Python virtual environment

As a recommendation, it is good practice to install and run the application on a virtual python environment. Such approach avoids possible issues with conflicting package versions.

We recommend you use pipenv:

    pip3 install pipenv

If you choose to install and run the Backend Client inside a virtual environment, please activate the environment before moving with the next steps.

For pipenv, a virtual environment is created and activated by:

    pipenv --three
    pipenv shell

Once you activate your environment your can safely install the python package under it.

For more information please refer to the tool's webpage.

Installing from PyPi

The Backend Client is available from PyPi and you can install the latest stable version with:

    pip install wirepas-backend-client

If you wish to install a particular version please see the release history from PyPi.

Installing from Github

First of all, make sure to clone the repository using the https address.

Checkout the git tag that corresponds to the release version you want to install and install the package with:

    pip3 install .

If you leave your branch checked out to the top of master, it is likely that you might run into a few issues. If you find them, please be so kind to raise them to us through an issue or bug report.

If you want to develop or patch a bug under your local environment, you can install the package in development mode through:

    pip3 install -e .

When installed in development mode, changes to the source files will be immediately visible.

Entrypoints

After installation, the Backend Client exposes several shell entrypoints that can be run from any location in your system. This section describes their usage in more detail.

Gateway command line interface

The Backend Client exposes an interactive gateway command line which is meant to interact with a gateway and sink pair connected to a given MQTT broker.

To start an interactive command line to interact with a broker located in your localhost environment type:

    wm-gw-cli

Connecting to a remote server requires you to specify the values as an argument or by reading from a yaml settings file. To view all the arguments, please issue the command:

    wm-gw-cli --help

Connecting to the MQTT broker

For the gateway client to function properly, it needs to establish a connection to the same MQTT broker where one or multiple gateways are connected to.

To do so, let's assume the following MQTT credentials are saved in ./examples/settings.yml:

 # examples/settings.yml
 # Example on how to set the MQTT credentials towards a local broker
 # with basic authentication
 mqtt_hostname: 127.0.0.1
 mqtt_password: password
 mqtt_username: username
 mqtt_port: 1883 # defaults to 8883 (secure port)
 mqtt_force_unsecure: True  # defaults to False (secure)

To start the gateway against the MQTT broker specified in ./examples/settings.yml, issue the following command:

    wm-gw-cli cli --settings ./examples/settings.yml

Once the client launches, you will be greeted with:

    Welcome to the Wirepas Gateway Client cli!
    Connecting to mosquittouser@127.0.0.1:1883 (unsecure: True)

    Type help or ? to list commands

    Type ! to escape shell commands
    Use Arrow Up/Down to navigate your command history
    Use CTRL-D or bye to exit

    09:50.17 | wm-gw-cli >

Commands

The gateway client has a set of commands which you can view by typing ? in the client's shell:

    wm-gw-cli > ?

    Documented commands (type help <topic>):
    ========================================
    EOF                     playback           set_loop_iterations
    bye                     q                  set_loop_timeout
    clear_offline_gateways  record             set_reply_greeting
    eof                     scratchpad_status  set_sink
    gateway_configuration   scratchpad_update  settings
    gateways                scratchpad_upload  shell
    help                    selection          sinks
    list                    send_data          toggle_byte_print
    ls                      set_app_config     toggle_pretty_print
    networks                set_config         track_devices
    nodes                   set_gateway        track_data_packets

For each available command you can get more help if you type ?command for example, to view how to send a datagram, type in the shell:

    wm-gw-cli > ?send_data

When typing commands, you can use the up and down arrows to browse your shell history. There is also support for running host shell commands, by prefacing the command with !.

Viewing connected devices

Upon a MQTT connection, the Backend Client will start populating an internal structure based on any incoming message. The topic is parsed and the information about the network devices are constructed.

The operation happens as messages are published from the network. It will take time to build a full map of the network, which will be faster depending on your network's packets per second.

In case you wish to view periodically which devices you have on your network, type in the following command:

    wm-gw-cli > ?track_devices

To exit the tracking loop, give a new line feed to the input stream and it will go back to the main command shell.

Device settings

When sending a downstream datagram, it is necessary to obey the Backend to Gateway API (WM-RM-128). For that reason, it is mandatory to know what is the target gateway identifier and in most cases the sink id as well.

Selecting the target gateway and sink

The wm-gw-cli provides an easy way to select the target gateway and sink. To make your choice, please type the following commands:

    wm-gw-cli > set_gateway
    0 : 2485378023427 : GatewayState.ONLINE
    1 : 2485378023426 : GatewayState.OFFLINE
    Please enter your gateway selection [0]: 0
    wm-gw-cli > set_sink
    0 : 3806491:2485378023427:sink0
    Please enter your sink selection [0]: 0

When setting the gateway, the wm-gw-cli shows you the following information object index : gateway id : state. Similarly, for the sink, the wm-gw-cli presents the object index : network : gateway id: sink id.

To make your selection, type the desired object index and press enter.

Once your selection is done, your commands will target that gateway id and sink id. If you wish to view what your current selection is, use the command:

    wm-gw-cli > selection

:warning: WARNING :warning:

The wm-gw-cli prompt will update itself with the current gateway and sink id. Throughout this document, that information has been edited out of the prompt.

Set app config

Setting the app config requires that your gateway and sink are selected. In case you have not done the selection prior to this command, the wm-gw-cli will ask for them.

When setting the app config, you are setting three values on the target sink:

• Diagnostic interval: the interval of network diagnostics (0, 30, 60, 300, 600, 1800)

• App config: A payload of a given size distributed to all associated devices on the network;

• App config sequence: The sequence of the new app config (must be higher than the current one - otherwise the app config won't take effect, please refer to WM-RM-100).

The app config payload is given either as a set of binary data without trailing 0x or as a utf8 data. For example, to send the binary 0x10x40x5 you should type 010405. If you simply want to have a text string on your app config, you can for example type "sink by the window".

As an example, on the wm-gw-cli you would type:

    # Remember! no white spaces allowing in the payload!
    # If you want to toggle byte string to hex string in the
    # app_config_data, please see toggle_byte_print

    wm-gw-cli > set_app_config 10 sink_by_the_window 30
    answer <<
        gw_id: 2485378023427
        sink_id: sink0
        req_id: 12079559850249952277
        res: GatewayResultCode.GW_RES_OK
        sink_id: sink0
        current_ac_range_min: 2000
        current_ac_range_max: 8000
        min_ac: 2000
        max_ac: 8000
        max_mtu: 102
        min_ch: 1
        max_ch: 27
        hw_magic: 3
        stack_profile: 1
        app_config_max_size: 80
        are_keys_set: False
        firmware_version: [3, 4, 37, 0]
        node_role: 17
        node_address: 4193520
        network_address: 3806491
        network_channel: 22
        app_config_diag: 30
        app_config_seq: 10
        app_config_data:
        b'sink_by_the_window\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
        \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
        \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
        \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
        \x00\x00\x00'
        channel_map: 286331153
        started: True

The answer will carry the gateway result and present you the new app config.

Upload and process a scratchpad

The wm-gw-cli supports uploading a new scratchpad and asking the sink to take it into use. A full network OTAP is not currently supported with the wm-gw-cli. To initiate a network OTAP, you will have to send the corresponding datagrams as mentioned in the remote API documentation.

The scratchpad commands are available as:

• scratchpad_status
• scratchpad_update
• scratchpad_upload

:warning: WARNING :warning:

An incorrect scratchpad or an incorrect OTAP can damage your network permanently. Please use these commands with care.

WPE Viewer

The WPE viewer, wm-wpe-viewer, provides a simple way to check if a WPE service and consume location updates for a given network.

The WPE viewer entrypoint requires correct population of the following settings:

    #examples/settings.yml
    wpe_service_definition: ./mywpesettings.json
    wpe_network: 1092 #optional

The mywpesettings.json consists of a standard WPE client configuration file. Please refer to WPE's official documentation for more details on how to obtain and configure such file.

The wpe network is the network identifier on which your devices operate. If you leave the parameter unset, the entrypoint will only perform a ping to the backend specified under the json file.

WNT Viewer

The WNT viewer, wm-wnt-viewer, provides a simple way to check if a WNT service is reachable and to consume messages streamed by WNT on its realtime metadata websocket. These message contain network and node diagnostic information. Please refer to WNT's public API documentation for further information.

The WNT viewer entrypoint requires correct population of the following settings:

    #examples/settings.yml
    wnt_username: "wntuser"
    wnt_password: "98asuyd907171ehjmasd"
    wnt_hostname: "wnthost.com"

Provisioning server

The provisioning server, wm-provisioning-server, provides an example implementation of the server side of the provisioning protocol. It must be used with the provisioning_joining_node application of the SDK. Please refer to the provisioning reference manual for further information.

The Provisioning server entrypoint requires correct population of the following settings:

    #examples/settings.yml
    mqtt_hostname: mqtt_broker_address_or_ip
    mqtt_password: password
    mqtt_username: username
    mqtt_port: 1883 # defaults to 8883 (secure port)
    mqtt_force_unsecure: True  # defaults to False (secure)

    provisioning_config: "./myprovisioningconfig.yml"

The myprovisioningconfig.yml consists of a list of authorized nodes and their associated provisioning data.

Framework

Besides the useful entrypoints to interact with Wirepas services, the goal of the Backend Client is to provide you a framework around which you can build test cases to help you during your integration, research and development phases.

Structure

The framework or package is organized in several submodules:

1. api
   1. api/influx
   2. api/mqtt
   3. api/mysql
   4. api/wnt
   5. api/wpe
2. management
3. mesh
   1. mesh/interfaces
   2. mesh/interfaces/mqtt
   3. mesh/interfaces/remote_api
   4. mesh/interfaces/beacon_api
4. messages
5. test
6. tools
7. cli

API submodule

The api submodule contains the interfaces to connect and handle data coming in and out of Wirepas' services and other 3rd party services such as databases and message brokers.

Each API or interface folder is organized with connectors, handlers and settings. The connectors file contains classes to establish a connection to the target interface. The handlers contain classes to process streams of input and output data. The settings file allows for handling parameter translation from input to class arguments (if needed).

Management submodule

The management submodule contains files regarding the runtime management of the framework. In particular the daemon which simplifies the multiprocess aspect of the Backend Client.

Mesh submodule

The mesh submodule contains files related to how devices are mapped within the Backend Client. In addition to those, the submodule contains an interfaces folder which contains specializations on how to handle specific Wirepas's APIs, such as the Gateway to Backend API.

:pencil2: NOTE :pencil2:

In our redesign effort we are targeting to improve the device classes and implement remote API support. Please follow the framework development for more up to date information.

Messages submodule

The messages submodule consists of a collection of message handler or decoders that handle translation of known APDUS, based on the endpoint they are associated to.

:pencil2: NOTE :pencil2:

We are considering moving the message handlers into the overarching Wirepas Backend APIs project. Please follow the framework development for more up to date information.

Test submodule

The test submodule contains classes to handle internal task scheduling and scripts to address particular mesh use cases.

Tools submodule

The tools submodule contains a collection of files with classes that handle the acquisition of user input, setup of the logging interface and general purpose utility methods and classes.

Cli submodule

Currently still a single file, this upcoming submodule will contain a re-factor of the gateway client to allow easier customization and construction of custom command line interfaces.

Examples

To better understand the framework inner-working and how you can extend it, we recommend that you take a look at the examples we have available in the examples folder:

• mqtt_viewer.py: connects to a MQTT broker and decodes on the fly any message that is known to the framework (see cookbook for usage example)

• find_all_nodes.py: simple interaction with the Gateway to Backend API that enumerates and prints outs discovered devices

• influx_viewer.py: run custom queries or export date ranges from an influx database where WNT is storing data

• settings.yml: an example of a yaml file that with commented out settings

• provisioning_config.yml: an example of a yaml file for the provisioning server

Backend Client's entrypoints also act as a good way to guide your development:

• wm-gw-cli: code behind the gateway command line interface that exemplifies how to build requests to the sink, receive and send data packets

• wm-wnt-viewer: example on how to consume data streamed by a WNT backend

• wm-wpe-viewer: example on how to consume data streamed by a WPE backend

• wm-provisioning-server: an example implementation of the server side of the wirepas provisioning protocol

For an example on how to build use case test cases, please refer to the kpi_adv.py script.

Logging to fluentd

The Backend Client has integrated logging with fluentd through Python's logging facility.

Routing data to a fluentd host requires that you define the target host when executing a Backend Client script.

To configure the target host, tag and record for the stream, ensure that you configure the examples/settings.yml file with

    # tags stream with app.mesh
    fluentd_hostname: "myfluenthost"
    fluentd_record: mesh
    fluentd_tag: app

Building and running over Docker

Docker allows application to run on a sandbox containing all the dependencies needed to run and execute them. If you are not familiar with Docker, please refer to the official documentation at docker.com.

Dockerhub

Backend Client builds are available from dockerhub under the Backend Client registry.

The latest tag points to the current stable release, whereas the edge tag points to the top of master.

The latest tag is built automatically at dockerhub whenever this repository is tagged.

The edge tag is built after each single merge into master.

To pull the Backend Client image from dockerhub use:

    docker pull wirepas/backend-client:latest
    docker pull wirepas/backend-client:<tag>

Running with docker

As the container will have no access to your local environment, you will have to propagate the input parameters by mounting a local file inside the container,eg, examples/settings.yml.

The default image command will launch the gateway client with the settings present under /home/wirepas/vars/settings.yml (container path).

To run it with docker type

    docker run -it \
               -v $(pwd)/examples/settings.yml:/home/wirepas/backend-client/vars/settings.yml \
               --net=host \
               wirepas/backend-client \
               wm-gw-cli \
               --settings /home/wirepas/backend-client/vars/settings.yml \
               --debug_level=critical

:warning: WARNING :warning:

If you want to run against a MQTT running in your host (localhost), you must overlay the container over your host's network. To do so, you must set the docker run parameter --net=host.

Running with compose

To run the Backend Client using docker compose, drop or move the settings file in container/examples/settings.yml and start the service with:

    docker-compose container/docker-compose.yml up

By default this will start the MQTT viewer example.

If you wish to run the gateway command client you can do so with:

    docker-compose -f container/docker-compose.yml \
                   run \
                   backend-client wm-gw-cli \
                   --settings /home/wirepas/backend-client/vars/settings.yml

If you prefer alpine based images, please change slim to alpine.

Building the image locally

To build the image locally in the root of the repo type:

    docker build -f container/slim/Dockerfile -t backend-client .

Alternatively you can also build using the docker-compose.yml present in the root of the directory:

    docker-compose -f container/docker-compose.yml  build

Source documentation

The source documentation is found from read the docs.

Contributing

We welcome your contributions!

Please read the instructions on how to do it and please review our code of conduct.

License

Copyright 2019 Wirepas Ltd licensed under Apache License, Version 2.0 See file LICENSE for full license details.