Skip to main content
Python Software Foundation 20th Year Anniversary Fundraiser  Donate today!

An extension for storing data in HDFS

Project description

ckanext-hdfs - HDFS storing extension

ckanext-hdfs is an extension for enabling the file storage in HDFS - Hadoop Distributed File System.

This extension provides an ability to let users store a certain resource in HDFS, instead of the local file system.


  • JAVA_HOME and HADOOP_HOME need to be set correctly.


This extension was developed and tested under CKAN-2.7.3 and HADOOP-3.0.0


To install ckanext-hdfs:

  1. Activate your CKAN virtual environment, for example:

    . /usr/lib/ckan/default/bin/activate
  2. Install the ckanext-hdfs Python package into your virtual environment:

    pip install ckanext-hdfs
  3. Add hdfs setting in your CKAN config file (by default the config file is located at /etc/ckan/default/production.ini) as follows:

    ckan.plugins = hdfs <other-plugins>
    ckan.hdfs.storage_path = /ckan/data
  4. Restart CKAN. For example if you’ve deployed CKAN with Apache on Ubuntu:

    sudo service apache2 reload

Development Installation

To install ckanext-hdfs for development, activate your CKAN virtualenv and do:

git clone
cd ckanext-hdfs
python develop
pip install -r dev-requirements.txt

Running the Tests

To run the tests, do:

nosetests --nologcapture --with-pylons=test.ini

To run the tests and produce a coverage report, first make sure you have coverage installed in your virtualenv (pip install coverage) then run:

nosetests --nologcapture --with-pylons=test.ini --with-coverage --cover-package=ckanext.hdfs --cover-inclusive --cover-erase --cover-tests

Registering ckanext-hdfs on PyPI

ckanext-hdfs should be available on PyPI as If that link doesn’t work, then you can register the project on PyPI for the first time by following these steps:

  1. Create a source distribution of the project:

    python sdist
  2. Register the project:

    python register
  3. Upload the source distribution to PyPI:

    python sdist upload
  4. Tag the first release of the project on GitHub with the version number from the file. For example if the version number in is 0.0.1 then do:

    git tag 0.0.1
    git push --tags


This work was supported by Institute for Information & communications Technology Promotion (IITP) grant funded by the Korea government (MSIT) (No.2017-00253, Development of an Advanced Open Data Distribution Platform based on International Standards)

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for ckanext-hdfs, version 0.0.1
Filename, size File type Python version Upload date Hashes
Filename, size ckanext_hdfs-0.0.1-py2-none-any.whl (19.6 kB) File type Wheel Python version py2 Upload date Hashes View
Filename, size ckanext-hdfs-0.0.1.tar.gz (20.2 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page