Skip to main content

Archippe is a data persistence micro service for pelops. It uses influxdb to store incoming values and publishes the history a series upon request.

Project description

Archippe is a data persistence micro service for pelops. It uses influxdb to store incoming values and publishes the history a series upon request.

For example, archippe should store all values from topic \room\temperature. For this purpose a series with the same name is used in influxdb. To retrieve data a message must be sent to \dataservice\request\room\temperature. The message can either be a single float or a simple json structure. In the first case, the float value defines the time span in seconds (from=now()-floatvalue, to=now()). The json structure consists of two timestamps and group-by in seconds (optional): {"from": "2009-11-10T22:00:00Z", "to": "2009-11-10T23:00:00Z", "group-by": 60} (timestamp are inclusive: where t<=to and t>=from; time format is %Y-%m-%dT%H:%M:%S.%fZ, group-by must be integer). The result from the query will be published to \dataservice\response\room\temperature. It contains a list of a values and their timestamp that are available for the given period: [{"time": 10, "value": 0.1}, {"time": 11, "value": 0.2}, ...].

Request json-message:

{
    "from": "2009-11-10T22:00:00Z",
    "to": "2009-11-10T23:00:00Z",
    "group-by": 60  # optional (equal to 0)
}

Response json-message:

{
    "first": "2009-11-10T22:00:01Z",
    "last": "2009-11-10T22:59:23Z",
    "len": 49,  # entries in data list
    "topic": "/test/example",
    "version": 2,  # version of the response format
    "group-by": 0,
    "data": [
        {"time": "2009-11-10T22:00:01Z", "value": 17.98},
        {"time": "2009-11-10T22:01:50Z", "value": 13.98},
        {"time": "2009-11-10T22:03:00Z", "value": 11.98},
        ...
        {"time": "2009-11-10T22:59:23Z", "value": 20.0}
    ]
}

In pelops exists a HistoryAgent-class that implements a client side interaction with this dataservice. A usage example can be found in nikippe - the charts CircularChart and SequentialChart rely on HistoryAgent to keep track of old data.

Pelops Overview

Pelops Overview

Archippe is part of the collection of mqtt based microservices pelops. An overview on the microservice architecture and examples can be found at (http://gitlab.com/pelops/pelops).

For Users

Installation Core-Functionality

Prerequisites for the core functionality are:

sudo apt install python3 python3-pip

Install via pip:

sudo pip3 install archippe

To update to the latest version add --upgrade as prefix to the pip3 line above.

Install via gitlab (might need additional packages):

git clone git@gitlab.com:pelops/archippe.git
cd archippe
sudo python3 setup.py install

This will install the following shell scripts: * archippe

The script cli arguments are: * ‘-c’/’–config’ - config file (mandatory) * ‘–version’ - show the version number and exit

influxdb

Installation

Install influx database and client. For example in ubuntu to install them use:

sudo apt install influxdb influxdb-client

and start the influx client with ìnflux.

Configuration

We need to create a database (archippe), an admin user, and a non-admin user with write access to this database.

Within the influx client:

create database archippe
use archippe
create user admin with password 'supersecret' with all privileges
create user pelops with password 'secret'
grant all on archippe to pelops
exit

YAML-Config

A yaml [1] file must contain four root blocks:

  • mqtt - mqtt-address, mqtt-port, and path to credentials file credentials-file (a file consisting of the entry mqtt with two sub-entries mqtt-user, mqtt-password) [2]

  • logger - which log level and which file to be used

  • influx - influx-address, influx-port, and path to credentials file credentials-file (a file consisting of the entry influx with two sub-entries influx-user, influx-password) [3]

  • data-persistence

    • topics - list of topics that should be persisted and their types

    • prefix - prefix for each topic to request historic data

    • response - prefix for each topic to publish historic data

mqtt:
    mqtt-address: localhost
    mqtt-port: 1883
    credentials-file: ~/credentials.yaml
    log-level: INFO

logger:
    log-level: DEBUG
    log-file: archippe.log

data-persistence:
    influx:
        influx-address: homebase.w.strix.at
        influx-port: 8086
        credentials-file: ~/credentials.yaml
        database: archippe  # influx database
        log-level: INFO
    topics:  # list of topics that should be persisted
        - topic: /test/temperature
          type: float  # float, integer, string, boolean
        - topic: /test/humidity
          type: float  # float, integer, string, boolean
    topic-request-prefix: /dataservice/request  # prefix for each topic to request historic data
    topic-response-prefix: /dataservice/response  # prefix for each topic to publish historic data

systemd

  • add systemd example.

For Developers

Getting Started

This service consists of two classes DataPersistence and Topic. For each topic that should be peristet an instance of Topic is created in DataPersistence.

Changes in the yaml structure must be mirrored in archippe/schema.py. It is a json-schema that verifies the provided yaml.

MyInfluxDBClient

Wrapper for influxdb.InfluxDBClient. Takes care initializing and writing of single datapoints to the db.

Next to the raw connectivity, this client provides two methods: * write_point: Write a single measurement value to the database (wrapper for influxdb.write_points method). * write_points: Write a list of entries (timestamp, value) into the measurement of the database.

Todos

  • none currently planed

Misc

The code is written for python3 (and tested with python 3.5 on an Raspberry Pi Zero with Raspbian Stretch).

Merge requests / bug reports are always welcome.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

Archippe-0.2.1.tar.gz (12.6 kB view details)

Uploaded Source

File details

Details for the file Archippe-0.2.1.tar.gz.

File metadata

  • Download URL: Archippe-0.2.1.tar.gz
  • Upload date:
  • Size: 12.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: Python-urllib/3.6

File hashes

Hashes for Archippe-0.2.1.tar.gz
Algorithm Hash digest
SHA256 c1951cab9a485c42d5c69b4e9f48a40f28e6bb701f189fe346a14836cc6ef0ec
MD5 01327e99c952d9e0d8d83eacc783ba21
BLAKE2b-256 61a9581e2eb17373d46279e20e6d7f70dfeea51ea98c9f241fdaa7611f91a2e1

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page