An emulator for Python applications to help in the design of IoT deployments based on the edge computing model
Project description
WoTemu
An emulator for Python applications to help in the design of IoT deployments based on the edge computing model. It is focused on the Web of Things paradigm by offering extended support for applications programmed with WoTPy.
As an emulator, WoTemu demands significantly higher computation resources than simulators in the same domain, it is, however, able to run real code, which simplifies the design process and provides more meaningful insights into the target architecture.
The main output of an emulation experiment is an HTML report describing the observed behaviour of the stack. Please see a demo report here.
Design
This project leverages Docker Swarm Mode to offer simple horizontal scaling across heterogeneous nodes; this enables the emulation of scenarios with hundreds of actors.
The following image shows a high-level view of a simple emulation stack. This serves to illustrate the main design choices behind WoTemu.
- All communications for the supported protocols (HTTP, CoAP, Websockets and MQTT) go through the network gateways; these use NetEm to shape the traffic and emulate real network conditions. This redirection is based on iptables and is invisible to the application.
- All nodes report their metrics to a central Redis store in a periodic fashion; this will be later used to build the final HTML report.
- Thanks to the capabilities of Docker Swarm, multiple replicas of a single node can be created easily and will automatically recover from errors. Nodes could be interpreted as templates, while each replica is an actual real instance of the node.
- A Docker API proxy service based on Tecnativa/docker-socket-proxy is always deployed in a manager node to enable nodes to access stack metadata (e.g. container IDs of other nodes in the same network).
System requirements
- Python 3.6+
- Docker Engine 20.10.0+ (swap limit capabilities should be enabled)
- Docker Compose 1.27.0+
- WoTemu (install with pip:
pip install wotemu
)
Such a recent version of the Docker Engine is required to ensure that
cap_add
is supported in Swarm Mode.
Quickstart
Emulation experiments are represented by instances of wotemu.topology.models.Topology
.
The recommended workflow to run an experiment is as follows:
- Create a
Topology
. - Build the Compose file that describes the stack of that
Topology
. - Run the stack on the Docker Swarm cluster.
- Stop the stack after an arbitrary amount of time.
- Build the final report from the collected data contained in the central Redis store.
Please note that the commands in this section should be executed in a Swarm manager.
(Optional) Create a development Swarm cluster
There is a Vagrant configuration file (Vagrantfile
) in this repository that may be used to quickly create a Swarm cluster consisting of three virtual machines (one manager and two workers) for development and test purposes. All dependencies for WoTemu are installed in the provision stage.
Run vagrant up
in your host to create and provision the three guest machines. Please note that you must manually run the /vagrant/scripts/join-swarm.sh
script once in both worker1
and worker2
to join the swarm.
Describe the topology
Topologies can be defined in a Python file exposing a topology
function that takes no arguments and returns an instance of Topology
. The following is such an example:
from wotemu.enums import NetworkConditions
from wotemu.topology.models import (BuiltinApps, Network, Node, NodeApp,
NodeResources, Topology)
_SERVER_GIST = "https://gist.github.com/agmangas/94cc5c3d9d5dcb473cff774b3522bbb6/raw"
def topology():
network_3g = Network(
name="3g",
conditions=NetworkConditions.REGULAR_3G)
node_server = Node(
name="server",
app=NodeApp(path=_SERVER_GIST, http=True),
networks=[network_3g],
scale=1)
host_server = "{}.{}".format(
node_server.name,
network_3g.name)
app_reader = NodeApp(
path=BuiltinApps.READER,
params={
"servient_host": host_server,
"thing_id": "urn:wotemu:quickstart:thing"
})
node_reader = Node(
name="reader",
app=app_reader,
networks=[network_3g],
resources=NodeResources(mem_limit="150M"),
scale=4)
topology = Topology(nodes=[
node_server,
node_reader
])
return topology
There are two types of nodes here:
- The reader uses the
READER
built-in application, which takes a WoTPy Servient index URL (servient_host
) and Thing ID (thing_id
) as arguments and periodically reads all properties exposed by the Thing; this particular instance ofREADER
is connected to the server node. All replicas of reader are constrained to 150MB of memory. - The server uses a custom application defined in a remote HTTP URL that exposes a Thing with two properties.
Both nodes are connected in a network that uses the REGULAR_3G
network conditions. The four replicas of reader will periodically read both properties from the single replica of server on a channel that displays the typical latency and bandwidth of a 3G connection.
Applications
An application (i.e. the code run by a Node
) is a Python file that exposes an asynchronous app
function that takes at least three positional arguments:
Variable | Type | Description |
---|---|---|
wot |
wotpy.wot.wot.WoT |
WoTPy WoT entrypoint decorated and pre-configured by WoTemu. |
conf |
wotemu.config.EnvConfig |
Environment configuration that is currently active. |
loop |
asyncio.AbstractEventLoop |
Loop that is running the application. |
The path
parameter of a NodeApp
instance should point to an application. There are three distinct options when setting the value of path
:
- Using a WoTemu built-in application (e.g.
BuiltinApps.READER
). - Using a remote HTTP URL.
- Using a local file path.
Loading applications from the filesystem of a custom Docker image based on agmangas/wotemu is arguably the most versatile option. To that end, you may use the optional image
parameter in the Node
class (set to agmangas/wotemu:latest
by default).
Please note that although WoTemu does not impose any restrictions on what is actually executed in the application function, your code should follow the asynchronous I/O programming model to avoid blocking the main thread (there are some WoTemu background processes running in the loop that is passed as argument).
Deploy the stack
A Compose file describing the emulation experiment can be created automatically from a topology file using the wotemu compose
CLI command:
wotemu compose --path ./examples/quickstart.py
This stack may then be deployed to the Swarm cluster in the usual fashion:
docker stack deploy -c ./examples/quickstart.yml quickstart
Build the final report
Metrics such as network packets, interactions or system usage data points will be periodically collected while the stack is active. The emulation stack can be stopped when the user considers that enough time has passed to gather a significant amount of data for the experiment:
wotemu stop --compose-file ./examples/quickstart.yml --stack quickstart
It is necessary to stop the stack with
wotemu stop
instead ofdocker stack rm
to capture a final snapshot of the stack state and keep the Redis store online.
An HTML report containing useful insights into the behaviour of the emulation stack can be then generated with the following command.
wotemu report --out /report/ --stack quickstart
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file wotemu-0.6.0.tar.gz
.
File metadata
- Download URL: wotemu-0.6.0.tar.gz
- Upload date:
- Size: 1.8 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.1 importlib_metadata/4.3.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.61.0 CPython/3.7.7
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 99d9d59f9b2da237fb971fd94374542865000c4213648e9ff9ec0ace87d01182 |
|
MD5 | 236a1d319f1cdcfa21a45f14e79faa87 |
|
BLAKE2b-256 | aa3a22da5eea87ddd27ae565c862ef9943189482686109ce0a3e63a363984ee0 |
File details
Details for the file wotemu-0.6.0-py3-none-any.whl
.
File metadata
- Download URL: wotemu-0.6.0-py3-none-any.whl
- Upload date:
- Size: 1.8 MB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.1 importlib_metadata/4.3.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.61.0 CPython/3.7.7
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | b10e0d0a3013c10ecde43e812d18f21041adc0ca562fa5b8dee8bfea7140aed9 |
|
MD5 | c8b8a8021c1243b567a8bd75f6f01a37 |
|
BLAKE2b-256 | e9f3aadd521420acc5757792d3e16a7f1d5c63c27c86891b2e0a7e2fc78d0342 |