ckanext-hdfs - HDFS storing extension
ckanext-hdfs is an extension for enabling the file storage in HDFS - Hadoop Distributed File System.
This extension provides an ability to let users store a certain resource in HDFS, instead of the local file system.
Notes:
JAVA_HOME and HADOOP_HOME need to be set correctly.
Requirements
This extension was developed and tested under CKAN-2.7.3 and HADOOP-3.0.0
Installation
To install ckanext-hdfs:
Activate your CKAN virtual environment, for example:
. /usr/lib/ckan/default/bin/activate
Install the ckanext-hdfs Python package into your virtual environment:
pip install ckanext-hdfs
Add hdfs setting in your CKAN config file (by default the config file is located at /etc/ckan/default/production.ini) as follows:
ckan.plugins = hdfs <other-plugins> ckan.hdfs.storage_path = /ckan/data
Restart CKAN. For example if you’ve deployed CKAN with Apache on Ubuntu:
sudo service apache2 reload
Development Installation
To install ckanext-hdfs for development, activate your CKAN virtualenv and do:
git clone https://github.com/etri-odp/ckanext-hdfs.git cd ckanext-hdfs python setup.py develop pip install -r dev-requirements.txt
Running the Tests
To run the tests, do:
nosetests --nologcapture --with-pylons=test.ini
To run the tests and produce a coverage report, first make sure you have coverage installed in your virtualenv (pip install coverage) then run:
nosetests --nologcapture --with-pylons=test.ini --with-coverage --cover-package=ckanext.hdfs --cover-inclusive --cover-erase --cover-tests
Registering ckanext-hdfs on PyPI
ckanext-hdfs should be available on PyPI as https://pypi.python.org/pypi/ckanext-hdfs. If that link doesn’t work, then you can register the project on PyPI for the first time by following these steps:
Create a source distribution of the project:
python setup.py sdist
Register the project:
python setup.py register
Upload the source distribution to PyPI:
python setup.py sdist upload
Tag the first release of the project on GitHub with the version number from the setup.py file. For example if the version number in setup.py is 0.0.1 then do:
git tag 0.0.1 git push --tags
Acknowledgements
This work was supported by Institute for Information & communications Technology Promotion (IITP) grant funded by the Korea government (MSIT) (No.2017-00253, Development of an Advanced Open Data Distribution Platform based on International Standards)
Metadata
Release files for ckanext-hdfs 0.0.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| ckanext-hdfs-0.0.1.tar.gz | 20.2 kB | Without provenance | Details for ckanext-hdfs-0.0.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| ckanext_hdfs-0.0.1-py2-none-any.whl | Python 2 | none | any | Details for ckanext_hdfs-0.0.1-py2-none-any.whl |
Total release size: 39.8 kB
Release files / ckanext-hdfs-0.0.1.tar.gz
| Download URL | ckanext-hdfs-0.0.1.tar.gz |
|---|---|
| Size | 20.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
04af740b27640c046e174854ffa1bd5b2dbd59e0517574f786254ac923e41c5b
|
|
BLAKE2b-256 checksum How to use checksums |
f20f63c98aba00687d05be4d47202307acd2bb61c9d282b8f73b531f481afded
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.12.1 pkginfo/1.4.2 requests/2.11.1 setuptools/20.4 requests-toolbelt/0.8.0 tqdm/4.28.1 CPython/2.7.12
|
Release files / ckanext_hdfs-0.0.1-py2-none-any.whl
| Download URL | ckanext_hdfs-0.0.1-py2-none-any.whl |
|---|---|
| Size | 19.6 kB |
| Tags | Python 2 |
|
SHA-256 checksum How to use checksums |
5eb48caa61d5af82f556a95dd6c18805a7ddc1f5099a70894525ab06601f02ac
|
|
BLAKE2b-256 checksum How to use checksums |
e883a74fb96cb47a9c5fd678f1b200db63596d379ebf6932354e79ed57267179
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.12.1 pkginfo/1.4.2 requests/2.11.1 setuptools/20.4 requests-toolbelt/0.8.0 tqdm/4.28.1 CPython/2.7.12
|
Security for ckanext-hdfs 0.0.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.