Skip to main content

ckanext-hdfs - HDFS storing extension

ckanext-hdfs is an extension for enabling the file storage in HDFS - Hadoop Distributed File System.

This extension provides an ability to let users store a certain resource in HDFS, instead of the local file system.

Notes:

  • JAVA_HOME and HADOOP_HOME need to be set correctly.

Requirements

This extension was developed and tested under CKAN-2.7.3 and HADOOP-3.0.0

Installation

To install ckanext-hdfs:

  1. Activate your CKAN virtual environment, for example:

    . /usr/lib/ckan/default/bin/activate
  2. Install the ckanext-hdfs Python package into your virtual environment:

    pip install ckanext-hdfs
  3. Add hdfs setting in your CKAN config file (by default the config file is located at /etc/ckan/default/production.ini) as follows:

    ckan.plugins = hdfs <other-plugins>
    ckan.hdfs.storage_path = /ckan/data
  4. Restart CKAN. For example if you’ve deployed CKAN with Apache on Ubuntu:

    sudo service apache2 reload

Development Installation

To install ckanext-hdfs for development, activate your CKAN virtualenv and do:

git clone https://github.com/etri-odp/ckanext-hdfs.git
cd ckanext-hdfs
python setup.py develop
pip install -r dev-requirements.txt

Running the Tests

To run the tests, do:

nosetests --nologcapture --with-pylons=test.ini

To run the tests and produce a coverage report, first make sure you have coverage installed in your virtualenv (pip install coverage) then run:

nosetests --nologcapture --with-pylons=test.ini --with-coverage --cover-package=ckanext.hdfs --cover-inclusive --cover-erase --cover-tests

Registering ckanext-hdfs on PyPI

ckanext-hdfs should be available on PyPI as https://pypi.python.org/pypi/ckanext-hdfs. If that link doesn’t work, then you can register the project on PyPI for the first time by following these steps:

  1. Create a source distribution of the project:

    python setup.py sdist
  2. Register the project:

    python setup.py register
  3. Upload the source distribution to PyPI:

    python setup.py sdist upload
  4. Tag the first release of the project on GitHub with the version number from the setup.py file. For example if the version number in setup.py is 0.0.1 then do:

    git tag 0.0.1
    git push --tags

Acknowledgements

This work was supported by Institute for Information & communications Technology Promotion (IITP) grant funded by the Korea government (MSIT) (No.2017-00253, Development of an Advanced Open Data Distribution Platform based on International Standards)

Metadata

Release files for ckanext-hdfs 0.0.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ckanext-hdfs 0.0.1
File Size Uploaded Provenance Actions
ckanext-hdfs-0.0.1.tar.gz 20.2 kB Without provenance Details for ckanext-hdfs-0.0.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for ckanext-hdfs 0.0.1
File Interpreter ABI Platform Actions
ckanext_hdfs-0.0.1-py2-none-any.whl Python 2 none any Details for ckanext_hdfs-0.0.1-py2-none-any.whl

Total release size: 39.8 kB

Release files / ckanext-hdfs-0.0.1.tar.gz

Download URL ckanext-hdfs-0.0.1.tar.gz
Size 20.2 kB
Tags Source
SHA-256 checksum
How to use checksums
04af740b27640c046e174854ffa1bd5b2dbd59e0517574f786254ac923e41c5b
BLAKE2b-256 checksum
How to use checksums
f20f63c98aba00687d05be4d47202307acd2bb61c9d282b8f73b531f481afded
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/1.12.1 pkginfo/1.4.2 requests/2.11.1 setuptools/20.4 requests-toolbelt/0.8.0 tqdm/4.28.1 CPython/2.7.12

Release files / ckanext_hdfs-0.0.1-py2-none-any.whl

Download URL ckanext_hdfs-0.0.1-py2-none-any.whl
Size 19.6 kB
Tags Python 2
SHA-256 checksum
How to use checksums
5eb48caa61d5af82f556a95dd6c18805a7ddc1f5099a70894525ab06601f02ac
BLAKE2b-256 checksum
How to use checksums
e883a74fb96cb47a9c5fd678f1b200db63596d379ebf6932354e79ed57267179
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/1.12.1 pkginfo/1.4.2 requests/2.11.1 setuptools/20.4 requests-toolbelt/0.8.0 tqdm/4.28.1 CPython/2.7.12

Security for ckanext-hdfs 0.0.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.0.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page