bcdata 0.1.0.post1

Data BC Distribution Service, automated

British Columbia’s data distribution service, automated.

There is a wealth of British Columbia geographic information available as open data, but access to much of it is available only via a ‘Custom Download’ through the Data Distribution Service application - direct download urls are not available. Data analysis tasks requiring many inputs can be tedious to set up or replicate.

This Python module and CLI script enables relatively quick, scriptable downloads of BC geographic data.

Note

• this tool is for my convenience, it is in no way endorsed by the Province of Britsh Columbia or DataBC

• use with care, please don’t overload the service

• the download service seems to be ok with many download requests but failures may be unpredictable

• data are generally licensed as OGL-BC, but it is up to the user to check the licensing for any data downloaded

Installation

Install via pip:

$ pip install bcdata

Usage

The most basic usage requires:

• a valid email address (a required Distribution Service form input, the address is not otherwise used)

• url (or identifier portion of url path) for any DataBC Catalog record that includes a ‘Custom Download’ button.

For example, to order and download airport data, use either https://catalogue.data.gov.bc.ca/dataset/bc-airports or bc-airports

Python module

>>> import bcdata
>>> dl = bcdata.download('bc-airports', '[e-mail]')
>>> dl
/tmp/bcdata/unzipped_download_folder/GSR_AIRPORTS_SVW.gdb

Download times will vary based mainly on the size of your requested data. Expect about a minute for the smallest requests to complete. Individual downloads will time out after 2hours - this is long enough for datasets larger than the max size permitted on the web form.

CLI

The CLI uses the $BCDATA_EMAIL environment variable if available, otherwise an email address must be provided as an option.

$ bcdata --help
Usage: bcdata [OPTIONS] DATASET

  Download a dataset from BC Data Distribution Service

Options:
  --email TEXT       Email address. Default: $BCDATA_EMAIL
  -o, --output TEXT  Destination folder to write.
  -f, --format TEXT  Output file format. Default: FileGDB
  --help             Show this message and exit.

Common uses might look something like this:

$ bcdata --email [e-mail] bc-airports  # basic usage
$ export BCDATA_EMAIL=[e-mail]         # set a default email
$ bcdata bc-airports                                 # use default email
$ bcdata -o my_spots.gdb bc-airports                 # download to specified output location
$ bcdata bc-airports \                               # get airports as shapefile
    -f shp \
    -o bc_airports

Note that data are downloaded to specified folder. For above example, a bc_airports folder would be created in the current working directory and the individual shp, prj etc files would be found within.

Projections / CRS

Several projections are available on request from the Download Service, but this tool does not support this option. All data are downloaded as the default CRS (BC Albers, which should be EPSG:3005).

Use some other tool to reproject your downloads.

Development and testing

Note that tests require Fiona (and thus GDAL) to verify downloads. Using a virtualenv is probably a good idea.

macOS/Linux/etc

$ mkdir bcdata_env
$ virtualenv bcdata_env
$ source bcdata_env/bin/activate
(bcdata_env)$ git clone [e-mail]:smnorris/bcdata.git
(bcdata_env)$ cd bcdata
(bcdata_env)$ pip install -e .[test]
(bcdata_env)$ export BCDATA_EMAIL=[e-mail]
(bcdata_env)$ py.test

Windows

Development setup on Windows should be quite similar but installing Fiona on Windows can be more challenging.

Credits

• @ateucher for the correct POST url and syntax

• pyskel for the python packaging template