Skip to main content

An interactive command-line utility for downloading data from a CKAN data portal

Project description

ckan-downloader

An interactive command-line utility for downloading data from a CKAN data portal.

Provide a CSV with or specify some dataset IDs to start downloading all resources attached to those datasets into folders.

Installation

pip install ckan-downloader

Example usage

$ python -m ckan_downloader

CKAN Downloader 0.1.0

What is the data portal URL? 
> geoscience.data.qld.gov.au

Test connection to https://geoscience.data.qld.gov.au/api/action/site_read was successful.

Do you have a CSV with the dataset IDs to download? (y/n) 
> y

What is the CSV file path? 
> tests/test1.csv

Does this CSV have a header row? (y/n) 
> y

The name of the field/column containing the dataset IDs is needed. The options are:
   id, PID, Report Title
Which field has the IDs? 
> PID

Which directory should the downloads be saved in? 
> downloads

Starting dataset cr109373
Downloading CR109373 Report Geometry (https://geoscience.data.qld.gov.au/dataset/e2f7ae5f-e62a-403d-ba55-539074a5380c/resource/geo-doc363732-cr109373/download/%252FReport%25252f109373%25252fDocument%25252f363732%25252f109373.zip) to downloads/cr109373/109373.zip
Downloading WHOLE REPORT (https://gsq-horizon.s3-ap-southeast-2.amazonaws.com/QDEX/109373/cr_109373_1.pdf) to downloads/cr109373/cr_109373_1.pdf
Starting dataset cr108134
Downloading CR108134 Report Geometry (https://geoscience.data.qld.gov.au/dataset/61d06582-c2cf-48ce-a5de-162e71f38ab3/resource/geo-doc361522-cr108134/download/%252FReport%25252f108134%25252fDocument%25252f361522%25252f108134.zip) to downloads/cr108134/108134.zip
Downloading WHOLE REPORT (https://gsq-horizon.s3-ap-southeast-2.amazonaws.com/QDEX/108134/cr_108134_1.pdf) to downloads/cr108134/cr_108134_1.pdf

Future improvements

  • add command line options to skip interactive mode
  • search by spatial extent
  • include or skip certain filetypes
  • use progress bars

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for ckan-downloader, version 0.1.1
Filename, size File type Python version Upload date Hashes
Filename, size ckan_downloader-0.1.1-py3-none-any.whl (4.6 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size ckan-downloader-0.1.1.tar.gz (4.2 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page