Skip to main content
Join the official Python Developers Survey 2018 and win valuable prizes: Start the survey!


Project description


Data Package Pipelines processors for CKAN.


# clone the repo and install it with pip

git clone
pip install -e .


datapackage-pipelines-ckan contains several pipeline processors for working with CKAN.


A processor to retrieve metadata about a CKAN resource from a CKAN instance and add it as a datapackage resource.

run: ckan.add_ckan_resource
  resource-id: d51c9bd4-8256-4289-bdd7-962f8572efb0
  ckan-api-key: env:CKAN_API_KEY  # an env var defining a ckan user api key
  • ckan-host: The base url (and scheme) for the CKAN instance (e.g.
  • resource-id: The id of CKAN resource
  • ckan-api-key: Either a CKAN user api key or, if in the format env:CKAN_API_KEY_NAME, an env var that defines an api key. Optional, but necessary for private datasets.


A processor to save a datapackage and resources to a specified CKAN instance.

run: ckan.dump.to_ckan
  ckan-api-key: env:CKAN_API_KEY
  overwrite_existing: true
  push_resources_to_datastore: true
    name: test-dataset-010203
    state: draft
    private: true
    owner_org: my-test-org
  • ckan-host: The base url (and scheme) for the CKAN instance (e.g.
  • ckan-api-key: Either a CKAN user api key or, if in the format env:CKAN_API_KEY_NAME, an env var that defines an api key.
  • overwrite_existing: If true, if the CKAN dataset already exists, it will be overwritten by the datapackage. Optional, and default is false.
  • push_resources_to_datastore: If true, newly created resources will be pushed the CKAN DataStore. Optional, and default is false.
  • push_resources_to_datastore_method: Value is a string, one of ‘upsert’, ‘insert’ or ‘update’. This will be the method used to add data to the DataStore (see Optional, the default is ‘insert’.
  • dataset-properties: An optional object, the properties of which will be used to set properties of the CKAN dataset.

CKAN dataset from datapackage

The processor first creates a CKAN dataset from the datapackage specification, using the CKAN api `package_create <>`__. If the dataset already exists, and parameter overwrite_existing is True, the processor will attempt to update the CKAN dataset using `package_update <>`__. All existing resources and dataset properties will be overwritten.

CKAN resources from datapackage resources

If the CKAN dataset was successfully created or updated, the dataset resources will be created for each resource in the datapackage, using `resource_create <>`__. If datapackage resource are marked for streaming (they have the dpp:streamed=True property), resource files will be uploaded to the CKAN filestore. For example, remote resources may be marked for streaming by the inclusion of the stream_remote_resources processor earlier in the pipeline.

Additionally, if push_resources_to_datastore is True, the processor will push resources marked for streaming to the CKAN DataStore using `datastore_create <>`__ and `datastore_upsert <>`__.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
datapackage_pipelines_ckan-0.0.2b0-py2.py3-none-any.whl (10.8 kB) Copy SHA256 hash SHA256 Wheel py2.py3 Oct 31, 2017
datapackage-pipelines-ckan-0.0.2b0.tar.gz (93.6 kB) Copy SHA256 hash SHA256 Source None Oct 31, 2017

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page