investigraph 0.0.4

etl pipeline for investigations with follow the money data

investigraph

Research and implementation of an ETL process for a curated and up-to-date public and open-source data catalog of frequently used datasets in investigative journalism.

Using prefect.io for ftm pipeline processing

Documentation

Tutorial

installation

pip install investigraph

example datasets

There is a dedicated repo for example datasets that can be used as a Block within the prefect.io deployment.

deployment

docker

docker-compose.yml for local development / testing, use docker-compose.prod.yml as a starting point for a production setup. More instructions here

run locally

Clone repo first.

Install app and dependencies (use a virtualenv):

pip install -e .

After installation, investigraph as a command should be available:

investigraph --help

Quick run a local dataset definition:

investigraph run <dataset_name> -c ./path/to/config.yml

Register a local datasets block:

investigraph add-block -b local-file-system/investigraph-local -u ./datasets

Register github datasets block:

investigraph add-block -b github/investigraph-datasets -u https://github.com/investigativedata/investigraph-datasets.git

Run a dataset pipeline from a dataset defined in a registered block:

investigraph run ec_meetings

View prefect dashboard:

make server

test

make install
make test

supported by

Media Tech Lab Bayern batch #3