performanceplatform-collector: tools for sendingdata to the Performance Platform
Project description
A python command line tool that aggregates data from third party sources and pushes the result through to the Performance Platform (see http://alphagov.github.io/performanceplatform-documentation/ for more details). This tool will currently interface with Google Analytics and Pingdom to retrieve data.
Installation
Using pip
$ pip install performanceplatform-collector
From source
$ git clone https://github.com/alphagov/performanceplatform-collector.git $ cd performanceplatform-collector $ virtualenv venv $ source venv/bin/activate $ python setup.py install
Usage
pp-collector takes paths to various JSON files as arguments:
$ pp-collector -q [query file] -c [credentials file] -t [token file] -b [backdrop file]
There are also some optional command line arguments you can provide pp-collector:
--console-logging Rather than logging out to log/collector.log it will output all logs to stdout/err --dry-run When it comes to submitting the gathered data to the Performance Platform it will skip making the POST requests and instead log out the url, headers and body to your terminal.
Configuration
There are four configuration files that get injected into pp-collector and are the four required parameters:
Query, contains everything about what the collector will do during execution. It provides an entrypoint that pp-collector will execute and provide the query and options k-v pairs:
{ "entrypoint": "performanceplatform.collector.pingdom", "query": { "name": "govuk" }, "options": { }, "data-set": { "data-group": "my-data-group", "data-set": "my-data-set" } }
Token, this holds the bearer token to be used by this collector when POSTing to the Performance Platform:
{ "token": "some long hex value" }
Credentials, passes through any usernames, passwords, API keys etc… that are required to communicate to the third party service you desire.
Backdrop, where backdrop lives:
{ "url": "https://www.performance.service.gov.uk/data" }
For our deployment of performanceplatform-collector we pull in deployment from performanceplatform-collector-config repo. You will find much more indepth information about the collector configuration in the repos README.
Entrypoints
The following entrypoints are currently available:
performanceplatform.collector.ga performanceplatform.collector.ga.trending performanceplatform.collector.ga.realtime performanceplatform.collector.pingdom
Extending performanceplatform-collector
performanceplatform-collector can be extended to support new types of collector. To do so you’ll need to add new entrypoints. For each new type of collector create a file at:
performanceplatform/collector/mycollectortype/__init__.py
Inside that file add a main function which has the following signature:
main(credentials, data_set_config, query, options, start_at, end_at)
These arguments are all strings which are forwarded from the command line.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Hashes for performanceplatform-collector-0.1.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 575bb345722601356b5c04dcbdaeb05d63294cfc9379eba9eba08e5b654cf54d |
|
MD5 | 036ab5326b9bd3a12c715b4c7671ea70 |
|
BLAKE2b-256 | 9bd7c512d08a1d343bfceebfc62840862eb336b791e302d86ae8d9c245d45616 |