CKAN extension that adds a "Download all" button to a dataset
This CKAN extension adds a “Download all” button to datasets. This downloads a zip file containing all the resource files and a datapackage.json.
This zip file is a good way to package data for storing or sending, because:
- you keep all the data files together
- you include the documentation (metadata) - avoids the common problem of being handed some data files and not know anything about it or where to find info
- the metadata is machine-readable, so can be used by tools, software and in
automated workflows. For example:
- validating a series of data releases all meet a standard schema
- loading it into a database, using the column types and foreign key relations specified in the metadata
If the resource is pushed/xloaded to DataStore then the schema (column types) is also included in the datapackage.json file.
This extension uses a CKAN background job to create the zip every time a dataset is created or updated (or its data dictionary is changed). This suits CKANs where all files are uploaded - if the underlying data file changes without the CKAN URL changing, then the zip will not include the update (until something else triggers the zip to update).
(This extension is inspired by ckanext-packagezip, but that is old and relied on ckanext-archiver and IPipe.)
Designed to work with CKAN 2.7+
Ideally it is used in conjunction with DataStore and xloader (or datapusher), so that the Data Dictionary is included as a schema in the datapackage.json, to describe the column types.
To install ckanext-downloadall:
Activate your CKAN virtual environment, for example:
Install the ckanext-downloadall Python package into your virtual environment:
pip install ckanext-downloadall
Add downloadall to the ckan.plugins setting in your CKAN config file (by default the config file is located at /etc/ckan/default/production.ini). e.g.
ckan.plugins = downloadall
Restart the CKAN worker. For example if you’ve deployed it with supervisord:
sudo supervisorctl restart ckan-worker:ckan-worker-00
Restart CKAN server. For example if you’ve deployed CKAN with Apache on Ubuntu:
sudo service apache2 reload
6. Ensure the background job ‘worker’ process is running - see https://docs.ckan.org/en/2.8/maintaining/background-tasks.html#running-background-jobs
# Include additional fields from the dataset in the datapackage.json (e.g. # those defined in a ckanext-scheming schema) # (optional, space separated list). ckanext.downloadall.dataset_fields_to_add_to_datapackage = district county
There is a command-line interface:
Examples of use:
downloadall update-zip gold-prices downloadall update-all-zips
“All resource data” appears as a normal resource, instead of seeing a “Download All” button
You need to enable this extension in the CKAN config and restart the server. See the Installation section above.
ImportError: No module named datapackage
This means you have an older version of ckanapi, which is a dependency of ckanext-downloadall. Install a newer version.
OSError: [Errno 13] Permission denied: ‘/data/ckan/resources/c89’
You are trying to update zips from the command-line but running the tasks synchronously, rather than with the normal worker process. In this case you need to run it as the www-data user e.g.:
sudo -u www-data /usr/lib/ckan/default/bin/downloadall -c /etc/ckan/default/production.ini update-all-zips --synchronous
To install ckanext-downloadall for development, activate your CKAN virtualenv and do:
git clone https://github.com/davidread/ckanext-downloadall.git cd ckanext-downloadall python setup.py develop pip install -r dev-requirements.txt
Remember to run the worker (in a separate terminal):
paster --plugin=ckan jobs worker --config=/etc/ckan/default/development.ini
Running the Tests
To run the tests, do:
nosetests --nologcapture --with-pylons=test.ini
To run the tests and produce a coverage report, first make sure you have coverage installed in your virtualenv (pip install coverage) then run:
nosetests --nologcapture --with-pylons=test.ini --with-coverage --cover-package=ckanext.downloadall --cover-inclusive --cover-erase --cover-tests
Releasing a New Version of ckanext-downloadall
ckanext-downloadall is availabe on PyPI as https://pypi.org/project/ckanext-downloadall/. To publish a new version to PyPI follow these steps:
Update the version number in the setup.py file. See PEP 440 for how to choose version numbers.
Update the CHANGELOG.md with details of this release.
Make sure you have the latest version of necessary packages:
pip install --upgrade setuptools wheel twine
Create a source and binary distributions of the new version:
python setup.py sdist bdist_wheel && twine check dist/*
Fix any errors you get.
Upload the source distribution to PyPI:
twine upload dist/*
Commit any outstanding changes:
git commit -a git push
Tag the new release of the project on GitHub with the version number from the setup.py file. For example if the version number in setup.py is 0.0.1 then do:
git tag 0.0.1 git push --tags
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size ckanext_downloadall-0.1.0-py2-none-any.whl (32.3 kB)||File type Wheel||Python version py2||Upload date||Hashes View hashes|
|Filename, size ckanext-downloadall-0.1.0.tar.gz (31.8 kB)||File type Source||Python version None||Upload date||Hashes View hashes|
Hashes for ckanext_downloadall-0.1.0-py2-none-any.whl
Hashes for ckanext-downloadall-0.1.0.tar.gz