Operation-Pluto 0.1.1

Grab and rinse financial and economic data.

Operation Pluto is a pipeline set-up. It plumbs financial and economic data. Focused markets are Hong Kong, U.S. and China.

This data pipeline is organized in Luigi framework with Python.

Available Data

Currently connected data sources :

Hong Kong

• Census and Statistics Department

• The Hong Kong Association of Banks

• Hong Kong Government Bond Programme

• Hong Kong Monetary Authority

• Hang Seng Indexes Company

• Rating and Valuation Department

United States

• U.S. Bureau of Labor Statistics

• Federal Reserve System

• St. Louis Federal Reserve Economic Data

China

• ?

Master Data

• Holiday API

Pipeline Organization

• Crawl websites, back-fill past data, and construct file directories. All done as code.

• One table in data source corresponds to one target file.

• Pipeline task is stateful. Overwrite source file the least possible.

Prerequisites

• Python 3.5

• Luigi 2.7

Getting Started

Have Python 3.5 installed and clone this repository :

# Clone this repository
$ git clone https://github.com/hydra-lab/operation-pluto

Install Python dependencies :

# Installing with Conda may not work
$ pip install -r requirements.txt

Set up Luigi configuration file :

# Rename luigi.cfg.sample to luigi.cfg
$ mv luigi.cfg.sample luigi.cfg

Configure proxies in luigi.cfg if you’re behind any :

[proxies]
https = https://username:password@hostname:port/

Test the installation. New data should be extracted and parsed into folder test/data :

$ python -m luigi --module main RunMock --local-scheduler
$ ls test/data

High-level job orchestration is done in main.py. e.g. RunAll() is the wrapper class to initialize whole data directory and trigger all processing tasks. In production, tasks should be run on Luigi server. Because Luigi daemon will not run on Windows, simply run :

# Run Luigi server on http://localhost:8082
$ luigid
# Run task on Luigi server
$ python -m luigi --module main RunAll

Schedule pipeline to run periodically in Task Scheduler or cron. Set up run.sh on Windows :

# Script on Windows
start luigid
python -m luigi --module main RunAll
cmd "/c taskkill /IM "luigid.exe" /T /F"

License

This project is licensed under GNU Affero General Public License, Version 3.0. See LICENSE for full license text.