Skip to main content

An emulator for Python applications to help in the design of IoT deployments based on the edge computing model

Project description

WoTemu

Docker Cloud Build   Docker Image Size (latest semver)   Build Status   PyPI   PyPI - Python Version

An emulator for Python applications to help in the design of IoT deployments based on the edge computing model. It is focused on the Web of Things paradigm by offering extended support for applications programmed with WoTPy.

As an emulator, WoTemu demands significantly higher computation resources than simulators in the same domain, it is, however, able to run real code, which simplifies the design process and provides more meaningful insights into the target architecture.

The main output of an emulation experiment is an HTML report describing the observed behaviour of the stack. Please see a demo report here.

Design

This project leverages Docker Swarm Mode to offer simple horizontal scaling across heterogeneous nodes; this enables the emulation of scenarios with hundreds of actors.

The following image shows a high-level view of a simple emulation stack. This serves to illustrate the main design choices behind WoTemu.

Design diagram

  • All communications for the supported protocols (HTTP, CoAP, Websockets and MQTT) go through the network gateways; these use NetEm to shape the traffic and emulate real network conditions. This redirection is based on iptables and is invisible to the application.
  • All nodes report their metrics to a central Redis store in a periodic fashion; this will be later used to build the final HTML report.
  • Thanks to the capabilities of Docker Swarm, multiple replicas of a single node can be created easily and will automatically recover from errors. Nodes could be interpreted as templates, while each replica is an actual real instance of the node.
  • A Docker API proxy service based on Tecnativa/docker-socket-proxy is always deployed in a manager node to enable nodes to access stack metadata (e.g. container IDs of other nodes in the same network).

System requirements

Such a recent version of the Docker Engine is required to ensure that cap_add is supported in Swarm Mode.

Quickstart

Emulation experiments are represented by instances of wotemu.topology.models.Topology.

The recommended workflow to run an experiment is as follows:

  1. Create a Topology.
  2. Build the Compose file that describes the stack of that Topology.
  3. Run the stack on the Docker Swarm cluster.
  4. Stop the stack after an arbitrary amount of time.
  5. Build the final report from the collected data contained in the central Redis store.

Please note that the commands in this section should be executed in a Swarm manager.

(Optional) Create a development Swarm cluster

There is a Vagrant configuration file (Vagrantfile) in this repository that may be used to quickly create a Swarm cluster consisting of three virtual machines (one manager and two workers) for development and test purposes. All dependencies for WoTemu are installed in the provision stage.

Run vagrant up in your host to create and provision the three guest machines. Please note that you must manually run the /vagrant/scripts/join-swarm.sh script once in both worker1 and worker2 to join the swarm.

Describe the topology

Topologies can be defined in a Python file exposing a topology function that takes no arguments and returns an instance of Topology. The following is such an example:

from wotemu.enums import NetworkConditions
from wotemu.topology.models import (BuiltinApps, Network, Node, NodeApp,
                                    NodeResources, Topology)

_SERVER_GIST = "https://gist.github.com/agmangas/94cc5c3d9d5dcb473cff774b3522bbb6/raw"


def topology():
    network_3g = Network(
        name="3g",
        conditions=NetworkConditions.REGULAR_3G)

    node_server = Node(
        name="server",
        app=NodeApp(path=_SERVER_GIST, http=True),
        networks=[network_3g],
        scale=1)

    host_server = "{}.{}".format(
        node_server.name,
        network_3g.name)

    app_reader = NodeApp(
        path=BuiltinApps.READER,
        params={
            "servient_host": host_server,
            "thing_id": "urn:wotemu:quickstart:thing"
        })

    node_reader = Node(
        name="reader",
        app=app_reader,
        networks=[network_3g],
        resources=NodeResources(mem_limit="150M"),
        scale=4)

    topology = Topology(nodes=[
        node_server,
        node_reader
    ])

    return topology

There are two types of nodes here:

  • The reader uses the READER built-in application, which takes a WoTPy Servient index URL (servient_host) and Thing ID (thing_id) as arguments and periodically reads all properties exposed by the Thing; this particular instance of READER is connected to the server node. All replicas of reader are constrained to 150MB of memory.
  • The server uses a custom application defined in a remote HTTP URL that exposes a Thing with two properties.

Both nodes are connected in a network that uses the REGULAR_3G network conditions. The four replicas of reader will periodically read both properties from the single replica of server on a channel that displays the typical latency and bandwidth of a 3G connection.

Applications

An application (i.e. the code run by a Node) is a Python file that exposes an asynchronous app function that takes at least three positional arguments:

Variable Type Description
wot wotpy.wot.wot.WoT WoTPy WoT entrypoint decorated and pre-configured by WoTemu.
conf wotemu.config.EnvConfig Environment configuration that is currently active.
loop asyncio.AbstractEventLoop Loop that is running the application.

The path parameter of a NodeApp instance should point to an application. There are three distinct options when setting the value of path:

  • Using a WoTemu built-in application (e.g. BuiltinApps.READER).
  • Using a remote HTTP URL.
  • Using a local file path.

Loading applications from the filesystem of a custom Docker image based on agmangas/wotemu is arguably the most versatile option. To that end, you may use the optional image parameter in the Node class (set to agmangas/wotemu:latest by default).

Please note that although WoTemu does not impose any restrictions on what is actually executed in the application function, your code should follow the asynchronous I/O programming model to avoid blocking the main thread (there are some WoTemu background processes running in the loop that is passed as argument).

Deploy the stack

A Compose file describing the emulation experiment can be created automatically from a topology file using the wotemu compose CLI command:

wotemu compose --path ./examples/quickstart.py

This stack may then be deployed to the Swarm cluster in the usual fashion:

docker stack deploy -c ./examples/quickstart.yml quickstart

Build the final report

Metrics such as network packets, interactions or system usage data points will be periodically collected while the stack is active. The emulation stack can be stopped when the user considers that enough time has passed to gather a significant amount of data for the experiment:

wotemu stop --compose-file ./examples/quickstart.yml --stack quickstart

It is necessary to stop the stack with wotemu stop instead of docker stack rm to capture a final snapshot of the stack state and keep the Redis store online.

An HTML report containing useful insights into the behaviour of the emulation stack can be then generated with the following command.

wotemu report --out /report/ --stack quickstart

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

wotemu-0.6.0.tar.gz (1.8 MB view details)

Uploaded Source

Built Distribution

wotemu-0.6.0-py3-none-any.whl (1.8 MB view details)

Uploaded Python 3

File details

Details for the file wotemu-0.6.0.tar.gz.

File metadata

  • Download URL: wotemu-0.6.0.tar.gz
  • Upload date:
  • Size: 1.8 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.3.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.61.0 CPython/3.7.7

File hashes

Hashes for wotemu-0.6.0.tar.gz
Algorithm Hash digest
SHA256 99d9d59f9b2da237fb971fd94374542865000c4213648e9ff9ec0ace87d01182
MD5 236a1d319f1cdcfa21a45f14e79faa87
BLAKE2b-256 aa3a22da5eea87ddd27ae565c862ef9943189482686109ce0a3e63a363984ee0

See more details on using hashes here.

File details

Details for the file wotemu-0.6.0-py3-none-any.whl.

File metadata

  • Download URL: wotemu-0.6.0-py3-none-any.whl
  • Upload date:
  • Size: 1.8 MB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.3.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.61.0 CPython/3.7.7

File hashes

Hashes for wotemu-0.6.0-py3-none-any.whl
Algorithm Hash digest
SHA256 b10e0d0a3013c10ecde43e812d18f21041adc0ca562fa5b8dee8bfea7140aed9
MD5 c8b8a8021c1243b567a8bd75f6f01a37
BLAKE2b-256 e9f3aadd521420acc5757792d3e16a7f1d5c63c27c86891b2e0a7e2fc78d0342

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page