Skip to main content

Routinely collecting heterogeneous surveillance data into a coherent database

Project description

PANDEM - Source

Integrate heterogeneous publuc health surveillance data sources into a coherent data base for PANDEM-2 project.

Description

Identify, map and integrate multiple pandemic related data into a coherent pandemic-management database. Developed within the H2020 project PANDEM-2, Pandem-Source allows users to systematically capture, standardize and analyze data coming from international and national surveillance databases, participatory surveillance projects, social networks and mass media. This tool is focused on flexibility so adding new sources or variables can be easily done as is required during a pandemic episode.

Target users

Public health data experts and data managers needing integrate several surveillance data.

Approach

Data integration follow a semantic approach. Data sources need to be described using a Data Labelling Schema (DLS) file that specifies the acquisition chennel (URL, git, local file or script) the format and how to map the input data to Pandem-Source variables. If further customistions are needed, the user can define them using simple python scripts.

The DLS ensures each source and variable is properly documented knwing its origin, meaning and data quality.

Supported sources

Pandem-Source is designed to be flexible and extensible so new sources can be easily added by end users. In order to demontstrate this principle. The following sources are supported Out of the Box

Sources for indicators

Sources for referentials

User Interface - timeseries explorer

Time series

User Interface - Integration dashboard

Integration dashboard

Requirements

  • Python 3.7 or higher
  • R 3.6.3 or higher
  • Docker (optional for supporting machine learning article classification)

Installing from pip

If you want to customize the installation folder (defaulted to ~/.pandemsource) you need to set the environment variable PANDEM_HOME to a different folder

pip install pandem-source

python -m pandemsource setup --install

Installing for contributors

If you want to customize the installation folder (defaulted to ~/.pandemsource) you need to set the environment variable PANDEM_HOME to a different folder

git clone https://github.com/pandem2/pandem-source

cd pandem-source

make init

make install

source env/bin/activate

python -m pandemsource setup --install

Defining sources o monitor

PANDEM-Source comes with a list of predefined sources it can monitor, but they are not all enabled by default. In order to use them them you have to manually activate them with the 'setup' command.

See all avaiable sources

python -m pandemsource setup -h

Activating a particular source

# Activating ECDC covid19 dataset monitoring
python -m pandemsource setup --ecdc-covid19

# Activating preloaded 2023 functional excercise data
python -m pandemsource setup --pandem-2-2023-fx

Running Pandem-Source

  • Set the PANDEM_HOME variable to a local foder (only if you have customized the default installation folder)
export PANDEM\_HOME=your data folder here 
  • Running monitoring and dashboard
python -m pandemsource start -d 

Troubleshooting

If the command python -m pandemsource start -d --no-nlp does not work. Please make sure you've installed R 3.6.3 or higher. If R is properly installed, please make sure you've also installed required dependencies with:

(In your terminal)

sudo apt install libxml2-dev libsodium-dev libssl-dev libcurl4-openssl-dev libgdal-dev libfontconfig1-dev libharfbuzz-dev libfribidi-dev 

(In the R interpreter)

install.packages(c("epitweetr", "dplyr", "shiny", "plotly", "DT", "jsonlite", "httr", "XML", "ggplot2", "epitweetr", "reticulate", "seqinr", "readr"))

PANDEM 2

PANDEM-2 is a H2020 EU-funded project that aims to develop new solutions for efficient, EU-wide pandemic management. The goal of PANDEM-2 is to prepare Europe for future pandemics through innovations in training and to build capacity between EU member states responding to pandemics on a cross-border basis.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pandem-source-1.1.7.tar.gz (425.7 kB view details)

Uploaded Source

Built Distribution

pandem_source-1.1.7-py3-none-any.whl (653.8 kB view details)

Uploaded Python 3

File details

Details for the file pandem-source-1.1.7.tar.gz.

File metadata

  • Download URL: pandem-source-1.1.7.tar.gz
  • Upload date:
  • Size: 425.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.10.12

File hashes

Hashes for pandem-source-1.1.7.tar.gz
Algorithm Hash digest
SHA256 e5564009921838d98f2345c895cef48feb834badc31b32ef22ea848afa2b2d2e
MD5 bd25521102357c9acbd4ce28c3879b92
BLAKE2b-256 f7e679c6794cfe429bbe92e81c2735cc91f913e672c5bfa5a456f381863ded62

See more details on using hashes here.

File details

Details for the file pandem_source-1.1.7-py3-none-any.whl.

File metadata

  • Download URL: pandem_source-1.1.7-py3-none-any.whl
  • Upload date:
  • Size: 653.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.10.12

File hashes

Hashes for pandem_source-1.1.7-py3-none-any.whl
Algorithm Hash digest
SHA256 8e07ed96dd51fa68c012755e46b609c03306fb6d953fec938d5c5809a32d7170
MD5 563c506725d7e9e3fcf25a85b365423f
BLAKE2b-256 5307a3695b84a334c25486a54b1d035744e07306025a0505e64ed23632a8f883

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page