metsuri 0.5.0

Serial log collector and uploader

Metsuri, a serial logger

Metsuri is a base component in a serial logging system. It makes it easy to monitor hardware that has an output that can be attached with, for example, a USB serial interface.

As the monitoring hardware you will need something that runs the necessary Python components. For example, a recent RaspberryPi will work splendidly. 16GB SD card is enough, probably a lot smaller will work, as the system tries to upload the data to the cloud as soon as it is received.

The system comprises three main components, serial-logger, log-uploader and log-download. The serial-logger attaches to a serial line and logs the received data to a file, rotating files with requested intervals, log-uploader uploads the received events to AWS CloudWatch, and you download the logs from the desired internal from the cloud with log-download.

It is possible to just log the data locally, but AWS CloudWatch makes it simple to share the updated logs with a team of people. It is also a very secure way to store the logs if the loggers are installed in a remote location.

Install needed packages

% sudo apt-get update
% sudo apt-get install picocom supervisor python3-venv

Create Python env

% python3 -m venv ve
% . ve/bin/activate
(ve) % pip install metsuri-*.whl

Copy supervisor config

% cd supervisord
% sudo cp *.conf /etc/supervisor/conf.d

Currently log_uploader.conf needs to be modified. Modify the command-line to use correct serial device, log group and log stream for the device. Each device should have its own log stream.

If you wish to test the configuration, you can attach the USB serial and call

% sudo supervisorctl reload

which will start services. Both services should end up in RUNNING when observed with

% sudo supervisorctl status

In case of problems, logs can be found under /var/log/supervisor/

Setup connection configuration for AWS

The device should include credentials to allow logging into a CloudWatch log stream.

The ~/.aws/config should include the [default] section to point out the correct region for the CloudWatch target. Alternatively, you can specify, for example, AWS_PROFILE environment variable among others. See AWS documentation for details.

[default]
output = json
region = eu-central-1

By default, the credentials are found in ~/.aws/credentials, and the location can be overridden with AWS_SHARED_CREDENTIALS_FILE environment variable.

Example /home/pi/.aws/credentials below.

[default]
aws_access_key_id=foo
aws_secret_access_key=bar

Fetching the logs from CloudWatch

In the following example we gather all logs from log stream with names starting with example from the log group EXAMPLE-LOG-GROUP downloading them to files in
output-log-dir.

This method reflects the state of the device logs exactly, but can be heavy with lots of stream or log entries. Use --interval command-line option to limit the amount of data, or specify a narrow time window with --from and --to.

AWS_SHARED_CREDENTIALS_FILE=creds log-download EXAMPLE-LOG-GROUP example output-log-dir