eida-statistics-aggregator 0.2.0

EIDA nodes statistics aggregator

# EIDA Statistics aggregation

This project should provide unified statistics about EIDA nodes usage.

## Aggregating data

Each EIDA node prepares an aggregation of their logging file using the same script.

This aggregation result is sent to a central database through a webservice provided by a central node

### Install and execute

With pipenv, run

pipenv install pipenv shell python aggregator/eida_log_aggregation.py –help

For now, the log file from seiscomp is expected to be a list of JSON entries compressed with BZIP2.

### Exemples

Aggregate one bz2 seiscomp logfile:

python aggregator/eida_log_aggregation.py –node NIEP –output-file 20201102_NIEP fdsnws-requests.log.2020-11-02.bz2

Also available with stdin:

echo “fdsnws-requests.log.2020-11-02.bz2” | python aggregator/eida_log_aggregation.py –node NIEP –output-file 20201102_NIEP

You can also agregate several logfiles:

python aggregator/eida_log_aggregation.py –node NIEP –output-file 202011_NIEP fdsnws-requests.log.2020-11*.bz2

### Test

From the projet’s root directory run

pipenv install pipenv shell pytest

### Aggregation problems

#### The Count distinct problem

Some information requested by EIDA need to count distint occurences of information (an IP, a country). A naive approach counting distinct occurences on each day and each node can’t be used to count the distinct occurences at a global scale nor for another timewindow.

Enters HyperLogLog, an algorithm allowing to estimate occurences for different timeframe. hll is implemented in Python and PostgreSQL this is why this project uses both technologies.

#### Anonimization

We want to anonimize every data that can link to a person. This is why IP adresses are hashed using the same algorithm on each datacenter, in order to have consistant statistics.

## Ingesting data

A webservice receiving POST request and ingesting the result in a database

## Creating reports

This code create automatic reports from the database