luftdatenpumpe 0.10.0

Process live and historical data from luftdaten.info. Filter by station-id, sensor-id and sensor-type, apply reverse geocoding, store into TSDB and RDBMS databases, publish to MQTT or just output as JSON.

.. image:: https://img.shields.io/badge/Python-3.6,%203.7-green.svg
:target: https://pypi.org/project/luftdatenpumpe/

.. image:: https://img.shields.io/pypi/v/luftdatenpumpe.svg
:target: https://pypi.org/project/luftdatenpumpe/

.. image:: https://img.shields.io/github/tag/hiveeyes/luftdatenpumpe.svg
:target: https://github.com/hiveeyes/luftdatenpumpe

.. image:: https://assets.okfn.org/images/ok_buttons/od_80x15_red_green.png
:target: https://github.com/hiveeyes/luftdatenpumpe

.. image:: https://assets.okfn.org/images/ok_buttons/oc_80x15_blue.png
:target: https://github.com/hiveeyes/luftdatenpumpe

.. image:: https://assets.okfn.org/images/ok_buttons/os_80x15_orange_grey.png
:target: https://github.com/hiveeyes/luftdatenpumpe

|

##############
Luftdatenpumpe
##############


*****
About
*****
Process live and historical data from `luftdaten.info`_. Filter by station-id, sensor-id
and sensor-type, apply reverse geocoding, store into TSDB_ and RDBMS_ databases
(InfluxDB_ and PostGIS_), publish to MQTT_ or just output as JSON.

.. figure:: https://cdn.jsdelivr.net/gh/hiveeyes/luftdatenpumpe@master/doc/logo.svg
:target: https://github.com/hiveeyes/luftdatenpumpe
:height: 200px
:width: 200px


********
Features
********

1. Luftdatenpumpe_ acquires the measurement readings either from the livedata API
of `luftdaten.info`_ or from its archived CSV files published to `archive.luftdaten.info`.
To minimize impact on the upstream servers, all data gets reasonably cached.

2. While iterating the readings, it optionally filters on station-id, sensor-id or sensor-type
and restrains information processing to the corresponding stations and sensors.

3. Then, each station's location information gets enhanced by

- attaching its geospatial position as a Geohash_.
- attaching a synthetic real-world address resolved using the reverse geocoding service Nominatim_ by OpenStreetMap_.

4. Information about stations can be

- displayed on STDOUT or STDERR in JSON format.
- filtered and transformed interactively through jq_, the swiss army knife of JSON manipulation.
- stored into RDBMS_ databases like PostgreSQL_ using the fine dataset_ package.
Being built on top of SQLAlchemy_, this supports all major databases.
- queried using advanced geospatial features when running PostGIS_, please
follow up reading the `Luftdatenpumpe PostGIS tutorial <doc-postgis_>`_.

5. Measurement readings can be

- displayed on STDOUT or STDERR in JSON format, which allows for piping into jq_ again.
- forwarded to MQTT_.
- stored to InfluxDB_ and then
- displayed in Grafana_.


***********
Screenshots
***********
Display luftdaten.info (LDI) stations and measurements in Grafana.


Filtering
=========
- Filter by different synthesized address components and sensor type.
- Display measurements from filtered stations on Grafana Worldmap Panel.
- Display filtered list of stations with corresponding information in tabular form.
- Measurement values are held against configured thresholds so points are colored appropriately.

.. image:: https://community.hiveeyes.org/uploads/default/original/2X/f/f455d3afcd20bfa316fefbe69e43ca2fe159e62d.png
:target: https://weather.hiveeyes.org/grafana/d/9d9rnePmk/amo-ldi-stations-5-map-by-sensor-type


Popup label
===========
- Humanized label computed from synthesized OpenStreetMap address.
- Numeric station identifier.
- Measurement value, unit and field name.

.. image:: https://community.hiveeyes.org/uploads/default/original/2X/4/48eeda1a1d418eaf698b241a65080666abcf2497.png
:target: https://weather.hiveeyes.org/grafana/d/9d9rnePmk/amo-ldi-stations-5-map-by-sensor-type


********
Examples
********
Some example installations, usually running live data feeds through them.

.. todo:: Improve, add more dashboards from "weather.hiveeyes.org" and "vmm.hiveeyes.org".

Canonical dashboards
====================
- `Feinstaub Verlauf <https://weather.hiveeyes.org/grafana/d/ioUrPwQiz/luftdaten-info-verlauf>`_
- `Feinstaub Karte <https://weather.hiveeyes.org/grafana/d/AOerEQQmk/luftdaten-info-karte>`_

Labs
====
- `LDI Demo #1 » Stations by name, country and state <https://weather.hiveeyes.org/grafana/d/yDbjQ7Piz/amo-ldi-stations-1-select-by-name-country-and-state>`_
- `LDI Demo #2 » Cascaded stations <https://weather.hiveeyes.org/grafana/d/Oztw1OEmz/amo-ldi-stations-2-cascaded-stations>`_
- `LDI Demo #3 » Measurements by cascaded location selector <https://weather.hiveeyes.org/grafana/d/lT4lLcEiz/amo-ldi-stations-3-cascaded-measurements>`_
- `LDI Demo #4 » Find stations by sensor type <https://weather.hiveeyes.org/grafana/d/kMIweoPik/amo-ldi-stations-4-select-by-sensor-type>`_
- `LDI Demo #5 » Map by location and sensor type <https://weather.hiveeyes.org/grafana/d/9d9rnePmk/amo-ldi-stations-5-map-by-sensor-type>`_


********
Synopsis
********
::

# List stations
luftdatenpumpe stations --station=28,297 --reverse-geocode

# Store list of stations and metadata into RDBMS database (PostgreSQL), also display on STDERR
luftdatenpumpe stations --station=28,1071 --reverse-geocode --target=postgresql://luftdatenpumpe@localhost/weatherbase

# Store readings into InfluxDB
luftdatenpumpe readings --station=28,1071 --target=influxdb://luftdatenpumpe@localhost/luftdaten_info

# Forward readings to MQTT
luftdatenpumpe readings --station=28,1071 --target=mqtt://mqtt.example.org/luftdaten.info


For a full overview about all program options including meaningful examples,
you might just want to run ``luftdatenpumpe --help`` on your command line
or visit `luftdatenpumpe --help`_.


************
Installation
************
If you are running Python 3 already,
installing the program should be as easy as::

pip install luftdatenpumpe

At this point, you should be able to conduct simple tests like
``luftdatenpumpe stations`` as seen in the synopsis section above.
At least, you should verify the installation succeeded by running::

luftdatenpumpe --version

However, you might have to resolve some prerequisites so you want to follow
the detailed installation instructions at `install Luftdatenpumpe`_.


****************
Luftdaten-Viewer
****************
These installation instructions outline how to build a powerful and
user-friendly interactive GIS system on top of PostGIS, InfluxDB,
Grafana and Luftdatenpumpe.

This is for all readers who want to learn about how to setup the
whole system to build such beautiful and interactive data
visualization compositions of map-, graph- and other panel-widgets
like outlined in the "Examples" section.

- `Luftdaten-Viewer Applications`_
- `Luftdaten-Viewer Databases`_
- `Luftdaten-Viewer Grafana`_


*******
License
*******
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation; either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

You should have received a copy of the GNU Affero General Public License
along with this program; if not, see:
<http://www.gnu.org/licenses/agpl-3.0.txt>,
or write to the Free Software Foundation,
Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA


********************
Content attributions
********************
The copyright of particular images and pictograms are held by their respective owners, unless otherwise noted.

Icons and pictograms
====================
- `Water Pump Free Icon <https://www.onlinewebfonts.com/icon/97990>`_ from
`Icon Fonts <http://www.onlinewebfonts.com/icon>`_ is licensed by CC BY 3.0.



.. _doc-virtualenv: https://github.com/hiveeyes/luftdatenpumpe/blob/master/doc/virtualenv.rst
.. _doc-postgis: https://github.com/hiveeyes/luftdatenpumpe/blob/master/doc/postgis.rst


.. _luftdaten.info: https://luftdaten.info/
.. _Luftdatenpumpe: https://github.com/hiveeyes/luftdatenpumpe

.. _luftdatenpumpe --help: https://github.com/hiveeyes/luftdatenpumpe/blob/master/doc/usage.rst
.. _install Luftdatenpumpe: https://github.com/hiveeyes/luftdatenpumpe/blob/master/doc/setup/luftdatenpumpe.rst
.. _Luftdaten-Viewer Applications: https://github.com/hiveeyes/luftdatenpumpe/blob/master/doc/setup/ldview-applications.rst
.. _Luftdaten-Viewer Databases: https://github.com/hiveeyes/luftdatenpumpe/blob/master/doc/setup/ldview-databases.rst
.. _Luftdaten-Viewer Grafana: https://github.com/hiveeyes/luftdatenpumpe/blob/master/doc/setup/ldview-grafana.rst

.. _Erneuerung der Luftdatenpumpe: https://community.hiveeyes.org/t/erneuerung-der-luftdatenpumpe/1199

.. _The Hiveeyes Project: https://hiveeyes.org/

.. _OpenStreetMap: https://en.wikipedia.org/wiki/OpenStreetMap
.. _Nominatim: https://wiki.openstreetmap.org/wiki/Nominatim
.. _Geohash: https://en.wikipedia.org/wiki/Geohash
.. _dataset: https://dataset.readthedocs.io/
.. _SQLAlchemy: https://www.sqlalchemy.org/
.. _TSDB: https://en.wikipedia.org/wiki/Time_series_database
.. _RDBMS: https://en.wikipedia.org/wiki/Relational_database_management_system
.. _MQTT: http://mqtt.org/

.. _PostgreSQL: https://www.postgresql.org/
.. _PostGIS: https://postgis.net/
.. _InfluxDB: https://github.com/influxdata/influxdb
.. _Grafana: https://github.com/grafana/grafana

.. _jq: https://stedolan.github.io/jq/