smartmeter-datacollector 1.0.2

Smart Meter Data Collector

Smart Meter Data Collector

---

The smartmeter-datacollector tool is a Python3 software which allows you to continuously retrieve data from supported smart meters (listed below). The acquired values can be forwarded to one or more data sinks like a MQTT broker or logger.

This tool is an open source software funded by EKZ (Elektrizitätswerke des Kantons Zürich) and developed by Supercomputing Systems AG. The goal of this voluntary initiative is to create a tool that can read, decode and decrypt data (if necessary) from the local interface of various smart meters and convert it into human-readable and simple formats. This should eliminate the complexity of decoding DLMS and similar protocols for the user. The user has the data in a format, which he can easily use in various use-cases ranging from simple vizualization (see Method 1: Raspberry Pi image with demo) to integration into energy management algorithms or into custom home automation software.

The following smart meters are supported:

• Landis+Gyr E450:
  Data pushed by smart meter over CII interface (wired M-Bus). Encoded with HDLC and DLMS/COSEM.
  • Unencrypted data
  • Encrypted data
• Iskraemeco AM550:
  Data pushed by smart meter over P1 interface (DSMR). Encoded with HDLC (IEC 62056-46) and DLMS/COSEM.
  • Unencrypted data
  • Encrypted data

The following data sinks are implemented:

• MQTT (v3.1.1):
  • Encryption
    • Unencrypted connection
    • Encrypted connection
      • TLS
      • optional custom CA certificate
  • Authentication
    • Unauthenticated
    • Authenticated with username / password
    • Authenticated with client certificate
• Logger to STDOUT

smartmeter-datacollector is fully configurable through a .ini configuration file. The smartmeter-datacollector-configurator webinterface can help to create and modify the configuration.

---

• Smart Meter Data Collector
• How to install
  • Method 1: Raspberry Pi image with demo
  • Method 2: Python package
    • Python Requirements
    • Installation
  • Method 3: Debian package
    • Debian Requirements
    • Installation
• How to use
  • Configuration
    • Specification
    • smartmeter-datacollector-configurator
  • Run manually
    • Configuration
  • Run as a systemd service
    • Configuration
• How to develop
  • Development Requirements
  • Installation
    • Debian / Ubuntu
  • Checkout the code
  • Setup Development Environment
  • Custom Commands / Workflows
• Acknowledgements

How to install

There are different methods how to use smartmeter-datacollector.

1. Raspberry Pi image with demo
2. Python package
3. Debian package

Method 1: Raspberry Pi image with demo

For a very easy first time usage of smartmeter-datacollector we provide a Raspberry Pi image (based on Raspberry Pi OS) which contains the following parts:

• smartmeter-datacollector as a systemd service
• smartmeter-datacollector-configurator webinterface
• demo
  • mosquiotto as a local MQTT broker to publish the measurements from smartmeter-datacollector
  • Telegraf to collect measurements published by smartmeter-datacollector over MQTT and store them in InfluxDB
  • InfluxDB to store the measurements
  • Grafana to visualize the measurements

This setup allows a (first time) user to install the image on a Raspberry Pi and immediately get started with smartmeter-datacollector. The following steps are required:

1. Download the Raspberry Pi image from the latest release.
2. Install the image on a (micro)SD card, e.g. using Raspberry Pi Imager.
3. Prepare the Raspberry Pi.
   1. Push the (micro)SD card into the proper slot.
   2. Use an RJ-45 cable (or configure WiFi later) to connect to a network.
   3. Connect the power cable to the Micro USB port.
4. Open the smartmeter-datacollector-configurator webinterface using http://<Raspberry Pi IP>:8000/ in your favorite browser.
   1. Load the pre-defined configuration.
   2. Adjust the settings of the connected smart meter.
   3. Deploy the adjusted configuration.
   4. Restart the smartmeter-datacollector service.
5. Open Grafana using http://<Raspberry Pi IP>:3000/ in your favorite browser.
6. Observe the measurements from your smart meter while they arrive periodically.

See Wiki/Demo for more details on the provided Raspberry Pi image with the demo.

Method 2: Python package

smartmeter-datacollector can be installed as a Python3 package either from PyPi or manually using a released wheel.

The Python3 package does not contain any infrastructure to run smartmeter-datacollector in the background or to automatically start it during a boot sequence. If such infrastructure is required either see Method 3: Debian package or provide it yourself.

Python Requirements

• Python >= 3.7 (tested with 3.8)

Installation

Install the package either as global Python package or in a virtualenv with

python3 -m pip install smartmeter-datacollector

Similarly the smartmeter-datacollector-configurator webinterface can be installed with

python3 -m pip install smartmeter-datacollector-configurator

Method 3: Debian package

smartmeter-datacollector is also available as a Debian (.deb) package from the releases which installs it system wide. The Debian package includes a systemd service file which enables smartmeter-datacollector to automatically start after booting the system.

Debian Requirements

• Distribution
  • Debian
• Release
  • Buster
• CPU architecture
  • amd64
  • armhf

Installation

Download the Debian package from releases and install it with

sudo apt install ./python3-smartmeter-datacollector_*.deb

Similarly the smartmeter-datacollector-configurator webinterface can be installed with a Debian package from its releases with

sudo apt install ./python3-smartmeter-datacollector-configurator_*.deb

How to use

The usage of smartmeter-datacollector depends on the installation method. Independent of the installation method a .ini configuration file is required to properly run smartmeter-datacollector.

Configuration

Specification

The .ini configuration file supports the following format

[reader0]
# type of the connected smart meter
type = lge450
# serial port to which the smart meter is connected
port = /dev/ttyUSB0
# optional encryption key
key = 

[sink0]
# type of the sink
type = mqtt
# host / IP address of the MQTT broker
host = localhost
# port of the MQTT broker
port = 1883
# whether to use TLS (Transport Layer Security)
tls = False
# optional path to the CA certificate used to validate the MQTT broker's certificate
ca_file_path = 
# whether to check the hostname of the MQTT broker against the certificate
check_hostname = False
# optional username for authentication with the MQTT broker
username = 
# optional password for authentication with the MQTT broker
password = 
# optional path to the client certificate for authentication with the MQTT broker
client_cert_path = 
# optional path to the private key for the client certificate for authentication with the MQTT broker
client_key_path = 

[sink1]
# type of the sink
type = logger
# name of the logger
name = DataLogger

[logging]
# log level configuration: DEBUG / INFO / WARNING / ERROR
default = WARNING
smartmeter = WARNING
collector = WARNING
sink = WARNING

There can be multiple [readerX] and [sinkY] sections for every connected smart meter and / or data sink.

See Wiki/Configuration for more details on the available configuration options.

smartmeter-datacollector-configurator

To simplify the process of generating a valid .ini configuration for smartmeter-datacollector the companion smartmeter-datacollector-configurator webinterface can be used. It supports

• a graphical approach to manage the configuration
• input validation to avoid invalid configurations
• loading / saving / discarding a configuration
• restarting smartmeter-datacollector (only if installed as a Debian package)

See Wiki/smartmeter-datacollector-configurator for more details on the webinterface and the configuration possibilities.

Run manually

Run smartmeter-datacollector with the following command:

smartmeter-datacollector --config datacollector.ini

The following command line arguments are supported:

• -h, --help: Shows the help output of smartmeter-datacollector.
• -c, --config CONFIG: Path to the .ini configuration file.
• -s,--saveconfig: Cteate a default .ini configuration file.
• -d, --dev: Enable development mode.

Configuration

The configuration file can be located anywhere and use any filename. If no .ini configuration file is specified a default configuration with the following options is used:

• Landys+Gir E450 smart meter in unencrypted mode connected to /dev/ttyUSB0
• LOGGER sink
• MQTT sink connected to a local broker without encryption or authentication

Run as a systemd service

When smartmeter-datacollector has been installed as a Debian package it provides a systemd .service file named python3-smartmeter-datacollector.service. Therefore the service can be managed using the systemctl command:

sudo systemctl start|stop|restart python3-smartmeter-datacollector

Enabling or disabling the service can also be managed using the systemctl command:

sudo systemctl enable|disable python3-smartmeter-datacollector

Inspecting the log messages can be done using the journalctl command:

journalctl -u python3-smartmeter-datacollector [-f]

Configuration

The systemd service expectes the .ini configuration to be located at /var/lib/smartmeter-datacollector/datacollector.ini. This is also the default location used by the smartmeter-datacollector-configurator webinterface to load and save the managed configuration.

How to develop

Help from the community for the smartmeter-datacollector project is always welcome. Please follow the next few chapters to setup a working development environment.

Development Requirements

• Python >= 3.7
• pipenv
• Optional software packages (Debian / Ubuntu)
  • python3-all
  • debhelper
  • dh-python
  • dh-systemd

Installation

Debian / Ubuntu

To install the listed minimal requirements run the following command:

sudo apt install git python3 pipenv

To install the optional requirements also run the following command:

sudo apt install python3-all debhelper dh-python dh-systemd

Checkout the code

Use git to clone / checkout smartmeter-datacollector from GitHub using

git clone https://github.com/scs/smartmeter-datacollector.git

Setup Development Environment

smartmeter-datacollector uses pipenv to manage its dependencies and setup a virtual environment. Run the following command to setup the initial development environment:

pipenv install --dev

This will install all runtime and development dependencies for smartmeter-datacollector in a new virtual environment. Now you are ready to start working on smartmeter-datacollector.

Custom Commands / Workflows

smartmeter-datacollector offers a few custom pipenv run commands to simplify certain development workflows:

• format_check checks if the code follows the autopep8 code formatting rules.
• format automatically adjusts the code to follow the autopep8 code formatting rules.
• isort_check checks if the order of the import statements is correct using isort.
• isort automatically re-orders the import statements using isort.
• lint_check checks if the code follows the pylint rules defined in pyproject.toml.
• lint automatically adjust the code to follow the pylint rules defined in pyproject.toml.
• license makes sure every Python (*.py) file contains the proper license header.
• build builds a Python package which can be uploaded to PyPI using twine.
• build_check uses twine to check if the built Python package will be accepted by PiPI.
• setup_check checks whether the dependencies defined in Pipfile / Pipfile.lock are in sync with setup.py.
• setup synchronizes the dependencies defined in Pipfile / Pipfile.lock with setup.py.
• debianize creates a debian/ directory used to build Debian source / binary packages.
• build_srcdeb builds a Debian source package which can be used to build a Debian (binary) package for any platform (e.g. using pbuilder)
• build_deb builds a Debian package for the current development plattform.

Make sure to run format_check / format, isort_check / isort, lint_check / lint, license, setup_check / setup before commiting changes to the repository to avoid unnecessary development cycles. smartmeter-datacollector uses GitHub Actions to check if these rules apply.

Acknowledgements

smartmeter-datacollector and its companion project smartmeter-datacollector-configurator have been developed by Supercomputing Systems AG on behalf of and funded by EKZ.