Skip to main content

Process data from live API of luftdaten.info. Filter by station- and sensor-id, apply reverse geocoding, output as JSON, store into RDBMS databases and publish to MQTT.

Project description

https://img.shields.io/badge/Python-2.7,%203.6-green.svg https://img.shields.io/pypi/v/luftdatenpumpe.svg https://img.shields.io/github/tag/hiveeyes/luftdatenpumpe.svg https://assets.okfn.org/images/ok_buttons/od_80x15_red_green.png https://assets.okfn.org/images/ok_buttons/oc_80x15_blue.png https://assets.okfn.org/images/ok_buttons/os_80x15_orange_grey.png

Luftdatenpumpe

https://cdn.jsdelivr.net/gh/hiveeyes/luftdatenpumpe@master/doc/logo.svg

About

  1. Luftdatenpumpe acquires the measurement readings either from the livedata API of luftdaten.info or from its archived CSV files published to archive.luftdaten.info.

  2. While iterating the readings, it optionally applies a filter based on station-id, sensor-id or sensor-type and collects information about all stations and sensors they are originating from.

  3. Then, each station’s location information gets enhanced by

    • attaching its geospatial position as a Geohash.

    • attaching a synthetic real-world address resolved using the reverse geocoding service Nominatim by OpenStreetMap.

  4. Information about stations can be

    • displayed on STDOUT or STDERR in JSON format.

    • filtered and transformed interactively through jq, the swiss army knife of JSON manipulation.

    • stored into RDBMS databases like PostgreSQL using the fine dataset package. Being built on top of SQLAlchemy, this supports all major databases.

  5. Measurement readings can be

    • displayed on STDOUT or STDERR in JSON format, which allows for piping into jq again.

    • forwarded to MQTT.

    • stored to InfluxDB and then

    • displayed in Grafana.

Screenshot

Display luftdaten.info (LDI) measurements on Grafana Worldmap Panel.

https://community.hiveeyes.org/uploads/default/original/2X/f/f455d3afcd20bfa316fefbe69e43ca2fe159e62d.png

Map and table display. Filter by synthesized address components and sensor type.

https://community.hiveeyes.org/uploads/default/original/2X/4/48eeda1a1d418eaf698b241a65080666abcf2497.png

Long name from OSM address and station id on overlay.

Demo

Live Data

List of stations

Synopsis

Overview

# List stations
luftdatenpumpe stations --station=28,297 --reverse-geocode

# Write list of stations and metadata to PostgreSQL database
luftdatenpumpe stations --station=28,1071 --reverse-geocode --target=postgresql:///luftdaten_meta

# Forward readings to MQTT
luftdatenpumpe readings --station=28,1071 --target=mqtt://mqtt.example.org/luftdaten.info

Details

$ luftdatenpumpe --help

Usage:
  luftdatenpumpe stations [options] [--target=<target>]...
  luftdatenpumpe readings [options] [--target=<target>]...
  luftdatenpumpe --version
  luftdatenpumpe (-h | --help)

Options:
  --source=<source>             Data source, either "api" or "file://" [default: api].
  --station=<stations>          Filter data by given location ids, comma-separated.
  --sensor=<sensors>            Filter data by given sensor ids, comma-separated.
  --reverse-geocode             Compute geographical address using the Nominatim reverse geocoder
  --target=<target>             Data output target
  --progress                    Show progress bar
  --version                     Show version information
  --dry-run                     Skip publishing to MQTT bus
  --disable-nominatim-cache     Disable Nominatim reverse geocoder cache
  --debug                       Enable debug messages
  -h --help                     Show this screen


Station list examples:

  # Display metadata for given stations in JSON format
  luftdatenpumpe stations --station=28,1071 --reverse-geocode

  # Display metadata for given sensors in JSON format
  luftdatenpumpe stations --sensor=657,2130 --reverse-geocode

  # Display list of stations in JSON format made of value/text items, suitable for use as a Grafana JSON data source
  luftdatenpumpe stations --station=28,1071 --reverse-geocode --target=json.grafana.vt+stream://sys.stdout

  # Display list of stations in JSON format made of key/name items, suitable for use as a mapping in Grafana Worldmap Panel
  luftdatenpumpe stations --station=28,1071 --reverse-geocode --target=json.grafana.kn+stream://sys.stdout

  # Write list of stations and metadata to RDBMS database (PostgreSQL), also display on STDERR
  luftdatenpumpe stations --station=28,1071 --reverse-geocode --target=postgresql:///luftdaten_meta --target=json+stream://sys.stderr

  # Read station information from RDBMS database (PostgreSQL) and format for Grafana Worldmap Panel
  luftdatenpumpe stations --source=postgresql:///luftdaten_meta --target=json.grafana.kn+stream://sys.stdout


Live data examples (InfluxDB):

  # Store into InfluxDB running on "localhost"
  luftdatenpumpe readings --station=28,1071 --target=influxdb://localhost:8086/luftdaten_info

  # Store into InfluxDB, with UDP
  luftdatenpumpe readings --station=28,1071 --target=udp+influxdb://localhost:4445/luftdaten_info

  # Store into InfluxDB, with authentication
  luftdatenpumpe readings --station=28,1071 --target=influxdb://username:password@localhost:8086/luftdaten_info


Archive data examples (InfluxDB):

  # Mirror archive of luftdaten.info
  wget --mirror --continue http://archive.luftdaten.info/ --limit-rate=1.5M
  wget --mirror --continue http://archive.luftdaten.info/ --accept-regex='2018-0[6789]'
  wget --mirror --continue --no-host-directories --directory-prefix=/var/spool/archive.luftdaten.info http://archive.luftdaten.info/

  # Ingest station information from CSV archive files, store into PostgreSQL
  luftdatenpumpe stations --source=file:///var/spool/archive.luftdaten.info --target=postgresql:///luftdaten_meta --reverse-geocode --progress

  # Ingest readings from CSV archive files, store into InfluxDB
  luftdatenpumpe readings --source=file:///var/spool/archive.luftdaten.info --station=483 --sensor=988 --target=influxdb://localhost:8086/luftdaten_info --progress


Live data examples (MQTT):

  # Publish data to topic "luftdaten.info" at MQTT broker running on "localhost"
  luftdatenpumpe readings --station=28,1071 --target=mqtt://localhost/luftdaten.info

  # MQTT publishing, with authentication
  luftdatenpumpe readings --station=28,1071 --target=mqtt://username:password@localhost/luftdaten.info


Combined examples:

  # Write stations to STDERR and PostgreSQL
  luftdatenpumpe stations --station=28,1071 --target=json+stream://sys.stderr --target=postgresql:///luftdaten_meta

  # Write readings to STDERR, MQTT and InfluxDB
  luftdatenpumpe readings --station=28,1071 --target=json+stream://sys.stderr --target=mqtt://localhost/luftdaten.info --target=influxdb://localhost:8086/luftdaten_info

Setup

Prerequisites

Debian packages:

apt install postgis redis-server redis-tools
Postgres database

Create database:

createuser --no-createdb --pwprompt hiveeyes
createdb --owner hiveeyes luftdaten_meta

Create read-only user:

su - postgres
psql

postgres=# \c luftdaten_meta
luftdaten_meta=# CREATE ROLE readonly WITH LOGIN PASSWORD 'readonly';
luftdaten_meta=# GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO readonly;
luftdaten_meta=# GRANT SELECT ON ALL TABLES IN SCHEMA public TO readonly;
Redis cache

This program extensively uses a runtime cache based on Redis. To make this work best, you should enable data durability with your Redis instance.

The append-only file is an alternative, fully-durable strategy for Redis. It became available in version 1.1. You can turn on the AOF in your Redis configuration file (e.g. /etc/redis/redis.conf):

appendonly yes

Python module

pip install luftdatenpumpe

References

Upstream luftdaten.info

Standing on the shoulders of giants

Development

License

This program is free software; you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program; if not, see: <http://www.gnu.org/licenses/agpl-3.0.txt>, or write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA

Content attributions

The copyright of particular images and pictograms are held by their respective owners, unless otherwise noted.

Icons and pictograms

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

luftdatenpumpe-0.5.0.tar.gz (28.4 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page