emews 0.9.3

Network traffic automation and prototyping.

eMews - Network Traffic Automation and Prototyping

eMews is a framework which automates client-side computer network traffic generation, while providing centralized monitoring of a network while traffic is being generated. eMews provides an API to write network applications which abstract away basic functionality such as peer-to-peer communication. These applications, called services, provide client-side traffic generation, but can also be used to prototype complex network protocols, or even agent-environment interaction, in which the environment is the network itself [2]. eMews is designed for emulated networks such as those built with CORE, enabling quick deployment over potentially large-scale networks which reside on a single physical host [1].

Bundled with eMews are services which can generate SSH and HTTPS client-side network traffic in a connection-orientated paradigm. This is different from a playback approach, in that the traffic generated factors in response traffic from the server, and can be captured as PCAP data representing valid TCP sessions. By writing your own eMews services, you can generate any client-side traffic for any network protocol you desire.

With eMews you can quickly deploy large-scale botnets for flooding-type network traffic captures. eMews comes with a botnet service which can generate distributed UDP or TCP traffic, and supports multiple concurrent botnets.

• eMews version used in IRI-18 [1] paper: v0.32.
• eMews version used in IRI-19 [2] paper: v0.4.

Quick Start

We will assume that eMews is going to be run with the CORE 9 network emulator on an Ubuntu 24.04 system. To install CORE, see here. While CORE has an Ubuntu package, it may be more straightforward to build CORE from source. eMews is tested to work with CORE 8 and 9, through CORE 9.2.1, the most recent version as of this writing. While this guide is written for CORE 9, if a step is different to perform in CORE 8, the difference will be mentioned so CORE 8 users can also work with eMews.

Create a Python virtual environment (venv) that you will install eMews in. The venv must be created using the venv tool, and the venv must be named '.venv'.

If you plan to write services for eMews, then a straightforward way to create a venv is to use an IDE such as PyCharm and create a new project for eMews development. By default, PyCharm will create a venv for the project (using the venv tool and with name '.venv'), and then you can enter it by simply opening a terminal session in PyCharm.

Assuming you have created and entered your virtual environment, download / install eMews using pip:

pip install emews

You need to know where eMews was installed so you can modify CORE's configuration accordingly. Type the following to get the eMews package path:

pip show emews

The path is located under Location in the output, for example:

Location: /home/username/emews/.venv/lib/python3.12/site-packages

Copy the path given at Location, which we will call <emews_pkg_path>, and open core.conf. In Ubuntu 24.04, core.conf is usually located at /opt/core/etc. Modify the following line in core.conf:

custom_services_dir = <emews_pkg_path>/emews/core_emu/core_services

If using CORE 8, you'll need to modify the following lines in core.conf instead:

custom_services_dir = <emews_pkg_path>/emews/core_emu/core8_legacy_services
custom_config_services_dir = <emews_pkg_path>/emews/core_emu/core8_config_services

eMews provides a sample core.conf file under <emews_pkg_path>/emews/core_emu/example_conf. eMews interfaces with CORE through CORE services which map to eMews nodes and services. These CORE services are generated by eMews and will be discussed in a bit. Note that CORE services are different from eMews services. Starting from CORE 8, CORE service structure changed and these services were called 'config services'. Legacy CORE services were still supported and called 'services'. With CORE 9, support for legacy services was dropped and 'config services' were renamed to 'services', including API changes which required updating eMews CORE service generation.

• The venv containing eMews does not need to be active when running a CORE network as CORE will know where eMews is located via the custom_services_dir entry in core.conf.

Building CORE Networks

CORE networks can be built using core_gui. When building a CORE network to run eMews, one CORE node in the network should be designated to run the eMews hub. This node should be the first one created in the network ('n1') and should be a host type like a PC. In core_gui, right-click on this node, and select 'Services'. If running CORE 8, you'll select 'Config Services' instead. Under 'Groups', select 'eMews' and then select 'HubNode' as the service. Note that multiple eMews services can run on a single CORE node.

• If you do not see 'eMews' as a group option, then most likely your custom_services_dir (or custom_config_services_dir if using CORE 8) was not set correctly in core.conf.

By default, the IP address assigned to the first node created in a CORE network is 10.0.0.20. This IP address is also where eMews expects the HubNode service to reside. If the IP address of the CORE node running the eMews hub is not 10.0.0.20, then this IP address will need to be set in the eMews system configuration file located at <emews_pkg_path>/emews/system.yml. Under the hub section of system.yml change this field:

address: 10.0.0.20

from 10.0.0.20 to the IP address of the CORE node running the eMews HubNode service.

• To skip having to change the hub node IP address in system.yml, simply assign the eMews 'HubNode' service to the first CORE node created.
• eMews was not originally designed with PyPi packaging in mind, and thus the global system configuration (along with services) are contained within the package.

All remaining CORE nodes that will run eMews services need to additionally run an eMews host. In core_gui, right-click on a CORE node which should be an eMews host, and select 'Services' (or 'Config Services' if using CORE 8). Select 'eMews' as the group, and select 'HostNode' as the service. Then select additional eMews services that you'd like to run on the selected node.

• The eMews hub can itself run eMews services without requiring the 'HostNode' service.

When running a CORE network with eMews-enabled nodes, you can both control eMews and monitor what it is doing by connecting to the hub node. In core_gui, you need to first enable a control network for your CORE network so you can connect to the eMews hub node externally. With your CORE network open in core_gui, click on the 'Session' menu, select 'Options', and type an IP address range for 'Control Network', for example 172.16.0.0/24.

• The IP address range for the control network should be private and not conflict with existing networks your physical host interfaces with, for example your school or home network. See here for a list of private IPv4 address ranges.

Example CORE networks running eMews nodes are located under <emews_pkg_path>/emews/core_emu/networks.

Controlling and Monitoring eMews

When running your CORE network, which is called a CORE session, you may want to run an eMews console and monitor to interact with eMews and view log output, respectively. To start a session of your network, click the 'Start Session' button, and wait for your network to finish start-up (for large networks this can take some time). Once your network is running, double-click on the hub node in core_gui. This opens a root shell, and in this shell type:

ifconfig

The NIC for the control network will be called ctrl0. Copy the IP address assigned to this NIC, which we will call <hub_ctrl_ip>. Then, open a local console, activate the eMews venv (or if using an IDE like PyCharm open the eMews project and then open a terminal session within the IDE) and type:

python3 -m emews -n monitor1 monitor --hub_address <hub_ctrl_ip>

This will create a local standalone eMews node called 'monitor1', which will connect to the hub node and output log messages. Now open another local console, activate the eMews venv as before (of if using an IDE open another terminal session) and type:

python3 -m emews -n console1 console --hub_address <hub_ctrl_ip>

This will create a local standalone eMews node called 'console1', which will connect to the hub node and provide a console in which you can type commands to eMews.

Console Node Commands

When a CORE session starts, eMews services will be suspended. To manually start all eMews services, type in the console node:

exp start

Here, 'exp' is short for 'experiment', which is what we refer to as eMews' goal, for example network traffic generation.

• eMews can start an experiment automatically by setting the system.yml field scenario_autostart to True under the hub section

To check if there are any issues with a running experiment, type:

exp status

If the status is green, then all eMews services are running normally. To get a detailed list of all services and their execution state, type:

list services

To stop all eMews services, type:

exp stop

The eMews console supports many commands to interact with eMews. You can check resource usage, start and stop individual services, and even create service instances on a node. Type help to get a list of commands. Type help <command> to get help on a specific command.

eMews Services

eMews comes with client-side services to interact with servers such as SSH and HTTPS, along with bots for generating various DDoS traffic. All services which eMews comes with are located here:

<emews_pkg_path>/emews/services

Each service is in a sub-folder named accordingly. Services are configured using .yml files within the service folder. There is at least one configuration per service called default.yml, but there may be others. When adding an eMews service to a CORE node in core_gui, the configuration loaded is default.yml. To load another configuration instead, right-click the CORE node and click 'Services'. Then under 'Selected', click the eMews services you would like to configure and click 'Configure'. All the available configurations are given in a drop-down list.

Helpful Hints

CORE network nodes are run as containers under root, and are built under /tmp/pycore.### (the '###' is a number). In this directory are directories for each CORE node, named <node_name>.conf, where <node_name> is the name of the node as assigned in core_gui (such as 'n1'). For example, the path to a CORE node called 'n1' would be:

/tmp/pycore.###/n1.conf

All CORE nodes running eMews contain log files which start with emews and end with .log. These files log console output (including exceptions) from each eMews daemon and service launcher, and may contain useful messages related to local issues. For example, a CORE node running an eMews host has two log files: emews_node_host.log and emews_servspawn.log. The first provides local (to the CORE node) log output of the eMews host, whereas the second provides local log output for eMews service spawning. To obtain eMews service log output among all nodes in the network, launch a monitor node.

If you want eMews services to start automatically when your CORE network is started, under <emews_pkg_path>/emews/system.yml, set hub.scenario_autostart to True. When your CORE network starts, eMews services will start automatically after an amount of seconds given by hub.scenario_autostart_delay. This enables you to launch a CORE network and start eMews services without having to connect a console session to the hub node.

References:

[1] B. Ricks, P. Tague and B. Thuraisingham, "Large-Scale Realistic Network Data Generation on a Budget," IEEE Intl Conference on Information Reuse and Integration for Data Science (IRI), Jul 2018

• URL: https://mews.sv.cmu.edu/papers/iri-18.pdf

[2] B. Ricks, B. Thuraisingham, and P. Tague, "Mimicking Human Behavior in Shared-Resource Computer Networks", IEEE Intl Conference on Information Reuse and Integration for Data Science (IRI), Jul 2019

• URL: https://mews.sv.cmu.edu/papers/iri-19.pdf