Skip to main content

A standalone web service that parses the contents of a CKAN site's data files and pushes them into its DataStore

Project description

Build Status Coverage Status Latest Version Downloads Supported Python versions Development Status License

DataPusher

DataPusher is a standalone web service that automatically downloads any CSV or XLS (Excel) data files from a CKAN site's resources when they are added to the CKAN site, parses them to pull out the actual data, then uses the DataStore API to push the data into the CKAN site's DataStore.

This makes the data from the resource files available via CKAN's DataStore API. In particular, many of CKAN's data preview and visualization plugins will only work (or will work much better) with files whose contents are in the DataStore.

To get it working you have to:

  1. Deploy a DataPusher instance to a server (or use an existing DataPusher instance)
  2. Enable and configure the datastore plugin on your CKAN site.
  3. Enable and configure the datapusher plugin on your CKAN site.

For details see the DataPusher documentation.

Note that if you installed CKAN using the package install option then a DataPusher instance should be automatically installed and configured to work with your CKAN site.

DataPusher is a replacement for DataStorer. It's built using CKAN Service Provider and Messytables.

The original author of DataPusher was Dominik Moritz dominik.moritz@okfn.org. For the current list of contributors see github.com/ckan/datapusher/contributors

Development

To install DataPusher for development:

git clone https://github.com/ckan/datapusher.git
cd datapusher
pip install -r requirements-dev.txt

To run the tests:

nosetests

To build the documentation:

pip install -r doc-requirements.txt
python setup.py build_sphinx

Releasing a New Version

To release a new version of DataPusher:

  1. Increment the version number in datapusher/init.py

  2. Build a source distribution of the new version and publish it to PyPI:

    python setup.py sdist bdist_wheel
    pip install --upgrade twine
    twine upload dist/*
    

    You may want to test installing and running the new version from PyPI in a clean virtualenv before continuing to the next step.

  3. Commit your setup.py changes to git, tag the release, and push the changes and the tag to GitHub:

    git commit setup.py -m "Bump version number"
    git tag 0.0.1
    git push
    git push origin 0.0.1
    

    (Replace both instances of 0.0.1 with the number of the version you're releasing.)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for datapusher, version 0.0.16
Filename, size File type Python version Upload date Hashes
Filename, size datapusher-0.0.16.tar.gz (9.1 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page