road-data-scraper 0.0.16

Scrapes and Cleans WebTRIS Traffic Flow API

Road Data Scraper

Scrapes and Cleans WebTRIS Traffic Flow API.

• Python Rewrite of ONS Road Data Pipeline written in R.
• Documentation of ONS Road Data Pipeline: https://datasciencecampus.github.io/road-data-pipeline-documentation/
• WebTRIS Traffic Flow API: https://webtris.highwaysengland.co.uk/api/swagger/ui/index

Developer Usage

Download and Install Python 3.9; if using Anaconda or Miniconda, create a virtual environment with Python 3.9, e.g., conda create --name py39 python=3.9

1. Git clone the repository: git clone https://github.com/dombean/road_data_scraper.git
2. Change Directory inside the road_data_scraper folder: cd road_data_scraper/
3. Install package in editable mode: pip install -e .
4. Change directory into package folder: cd src/road_data_scraper/
5. Adjust config.ini file accordingly
6. Run the script: python main.py or python3 main.py

Adjusting the Config File (config.ini)

There are 5 main configuration options in the config.ini file:

• start_date: provide a date in quotes, in the format, %Y-%m-%d; e.g, "2021-01-01" -- which is 1st January 2021.
• end_date: provide a date in quotes, in the format, %Y-%m-%d; e.g, "2021-01-31" -- which is 31st January 2021.
• test_run: can take on two values -- True or False. Set test_run=False, when you want to download the entire data set. test_run by default is set to True, this is just to check the Pipeline works correctly (this will run the entire Pipeline on a subset of the available URL's).
• generate_report: can take on two values -- True or False. By default, this is set to True, this will generate a HTML report with tables and graphs, showing the Active and Inactive ID's for each road sensor -- MIDAS, TMU, and TAME.
• output_path: provide a path, as a string, in order to save the outputs generated by the Road Data Scraper Pipeline; for example, "/home/user/Documents/"
• rm_dir: can take on two values -- True or False. Set rm_dir=True, when you have a Google Cloud VM Instance and you don't want to storage the data on the VM (assuming you set gcp_storage=True).

Google Cloud (GCP) Storage Options

Options to save output data to a Google Cloud bucket.

• gcp_storage: can take on two values -- True or False. Set gcp_storage=True. This will save the data generated by a run of the Pipeline to a Google Cloud bucket.
• gcp_credentials: provide a path to your GCP credentials json file -- as a string; for example, "/home/user/gcp_credentials.json".
• gcp_bucket_name: provide the name of the GCP bucket -- as a string; for example, "road_data_scraper_bucket".
• gcp_blob_name: provide the name of the folder, you want the Pipeline to save the data to, in the GCP bucket -- as a string; for example, "landing_zone".

Google Cloud VM Instance Setup

1. Login to Google Cloud Platform and click on Compute Engine in the left side-bar.
2. Then, in the left side-bar, click on Marketplace and search for Ubuntu 20.04 LTS (Focal), then, click LAUNCH.
3. Name the instance appropriately; click COMPUTE-OPTIMISED (note: leave the defaults -- 4 vCPU, 16 GB memory); under Firewall, click Allow HTTPS traffic; and finally CREATE the VM instance.
4. SSH into the VM instance.
5. Run the following commands: sudo apt-get update && sudo apt-get dist-upgrade -y && sudo apt-get install python3-pip -y && sudo apt-get install wget -y
6. Pip install the road_data_scraper Package using the command: pip install road_data_scraper
7. Upload GCP json credentials file.
8. Download the config.ini file using the command: wget https://raw.githubusercontent.com/dombean/road_data_scraper/main/src/road_data_scraper/config.ini
9. Download the runner.py file using the command: wget https://raw.githubusercontent.com/dombean/road_data_scraper/main/runner.py
10. Open runner.py and put in the absolute path to the config.ini file.
11. Change config.ini parameters accordingly, see README section: Adjusting the Config File (config.ini).
12. Run the Road Data Scraper Pipeline using the command: python3 runner.py

Google Cloud Run Setup

Note: Install Docker and Google Cloud SDK.

• Login to Google Cloud on the command line: gcloud auth login
• Configure Google Cloud Project on the command line: gcloud config set project <project-name>
• Configure Docker and Google Cloud Credentials: gcloud auth configure-docker

1. Git clone the repository: git clone https://github.com/dombean/road_data_scraper.git
2. Change Directory inside the road_data_scraper folder: cd road_data_scraper/
3. Download Google Cloud JSON Credentials into the repository.
4. Build the Docker Image: docker build -t road-data-scraper -f Dockerfile .
5. Test the Docker Image: docker run -it --env PORT=80 -p 80:80 road-data-scraper
6. Tag the Docker Image: docker tag road-data-scraper eu.gcr.io/<project-name>/road-data-scraper
7. Push the Docker Image: docker push eu.gcr.io/<project-name>/road-data-scraper
8. Deploy the Docker Image on Google Cloud Run: gcloud run deploy road-data-scraper --image eu.gcr.io/<project-name>/road-data-scraper --platform managed --region europe-west2 --timeout "3600" --cpu "4" --memory "16Gi" --max-instances "3"