Skip to main content

An interactive command-line utility for downloading data from a CKAN data portal

Project description

ckan-downloader

An interactive command-line utility for downloading data from a CKAN data portal.

Provide a CSV with or specify some dataset IDs to start downloading all resources attached to those datasets into folders.

Installation

pip install ckan-downloader

Example usage

$ python -m ckan_downloader

CKAN Downloader 0.1.0

What is the data portal URL? 
> geoscience.data.qld.gov.au

Test connection to https://geoscience.data.qld.gov.au/api/action/site_read was successful.

Do you have a CSV with the dataset IDs to download? (y/n) 
> y

What is the CSV file path? 
> tests/test1.csv

Does this CSV have a header row? (y/n) 
> y

The name of the field/column containing the dataset IDs is needed. The options are:
   id, PID, Report Title
Which field has the IDs? 
> PID

Which directory should the downloads be saved in? 
> downloads

Starting dataset cr109373
Downloading CR109373 Report Geometry (https://geoscience.data.qld.gov.au/dataset/e2f7ae5f-e62a-403d-ba55-539074a5380c/resource/geo-doc363732-cr109373/download/%252FReport%25252f109373%25252fDocument%25252f363732%25252f109373.zip) to downloads/cr109373/109373.zip
Downloading WHOLE REPORT (https://gsq-horizon.s3-ap-southeast-2.amazonaws.com/QDEX/109373/cr_109373_1.pdf) to downloads/cr109373/cr_109373_1.pdf
Starting dataset cr108134
Downloading CR108134 Report Geometry (https://geoscience.data.qld.gov.au/dataset/61d06582-c2cf-48ce-a5de-162e71f38ab3/resource/geo-doc361522-cr108134/download/%252FReport%25252f108134%25252fDocument%25252f361522%25252f108134.zip) to downloads/cr108134/108134.zip
Downloading WHOLE REPORT (https://gsq-horizon.s3-ap-southeast-2.amazonaws.com/QDEX/108134/cr_108134_1.pdf) to downloads/cr108134/cr_108134_1.pdf

Future improvements

  • add command line options to skip interactive mode
  • search by spatial extent
  • include or skip certain filetypes
  • use progress bars

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ckan-downloader-0.1.1.tar.gz (4.2 kB view hashes)

Uploaded Source

Built Distribution

ckan_downloader-0.1.1-py3-none-any.whl (4.6 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page