Harvester Next Generation Core for CKAN
Project description
Harvester Next Generation for CKAN
Install
pip install ckan-harvester
Use data.json sources
from harvester.data_json import DataJSON
dj = DataJSON()
dj.url = 'https://data.iowa.gov/data.json'
ret, error = dj.download_data_json()
print(ret, error)
# True None
ret, error = dj.load_data_json()
print(ret, error)
# True None
ret, errors = dj.validate_json()
print(ret, errors)
# False ['Error validating JsonSchema: \'bureauCode\' is a required property ...
# full dict with the source
print(dj.data_json)
"""
{
'@context': 'https://project-open-data.cio.gov/v1.1/schema/catalog.jsonld',
'@id': 'https://data.iowa.gov/data.json',
'@type': 'dcat:Catalog',
'conformsTo': 'https://project-open-data.cio.gov/v1.1/schema',
'describedBy': 'https://project-open-data.cio.gov/v1.1/schema/catalog.json',
'dataset': [{
'accessLevel': 'public',
'landingPage': 'https://data.iowa.gov/d/23jk-3uwr',
'issued': '2017-01-30',
'@type': 'dcat:Dataset',
...
"""
# just headers
print(dj.headers)
"""
{
'@context': 'https://project-open-data.cio.gov/v1.1/schema/catalog.jsonld',
'@id': 'https://data.iowa.gov/data.json',
'@type': 'dcat:Catalog',
'conformsTo': 'https://project-open-data.cio.gov/v1.1/schema',
'describedBy': 'https://project-open-data.cio.gov/v1.1/schema/catalog.json',
}
"""
for dataset in dj.datasets:
print(dataset['title'])
Impaired Streams 2014
2009-2010 Iowa Public School District Boundaries
2015 - 2016 Iowa Public School District Boundaries
Impaired Streams 2010
Impaired Lakes 2014
2007-2008 Iowa Public School District Boundaries
Impaired Streams 2012
2011-2012 Iowa Public School District Boundaries
Active and Completed Watershed Projects - IDALS
2012-2013 Iowa Public School District Boundaries
2010-2011 Iowa Public School District Boundaries
2016-2017 Iowa Public School District Boundaries
2014 - 2015 Iowa Public School District Boundaries
Impaired Lakes 2008
2008-2009 Iowa Public School District Boundaries
2013-2014 Iowa Public School District Boundaries
Impaired Lakes 2010
Impaired Lakes 2012
Impaired Streams 2008
Use CSW sources
from harvester.csw import CSWSource
c = CSWSource(url='http://data.nconemap.com/geoportal/csw?Request=GetCapabilities&Service=CSW&Version=2.0.2')
csw.connect_csw()
# True
csw_info = csw.read_csw_info()
print('CSW title: {}'.format(csw_info['identification']['title']))
# CSW title: ArcGIS Server Geoportal Extension 10 - OGC CSW 2.0.2 ISO AP
Development
To setup a develop environment, clone the repository and in a virtualenv install the dependencies
pip install -r requirements.txt
This will install the library in development mode, and other libraries for tests.
Test
Then to run the test suite with pytest:
pytest
We use pytest-vcr based on the wonderful VCRpy, to mock http requests. In this way, we don't need to hit the real internet to run our test (which is very fragile and slow), because there is a mocked version of a each response needed by tests, in vcr's cassettes format.
In order to update these cassettes just run as following:
pytest --vcr-record=all
To actually hit the internet without use mocks, disable the plugin
pytest --disable-vcr
Tests without a CKAN instance
python -m pytest tests
================ test session starts =============
platform linux -- Python 3.6.8, pytest-5.2.0, py-1.8.0, pluggy-0.13.0
rootdir: /home/hudson/dev/datopian/ckan-ng-harvester-core
plugins: vcr-1.0.2
collected 17 items
tests/test_csw_dataset_adapter.py .... [ 23%]
tests/test_data_json.py ....... [ 64%]
tests/test_datajson_dataset_adapter.py .....[100%]
=============== 17 passed in 17.52s ==============
Tests with a CKAN instance.
You will need to copy settings.py file to local_settings.py file and fill the required values.
You can use a local or remote CKAN instance.
python -m pytest tests_with_ckan/test_harvest.py
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for ckan_harvester-0.109-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6a98ff4316d8a649e233164941226945f1f4d36811517db7c846136db93df685 |
|
MD5 | 81e25359358edd38a4568ac16a456e1e |
|
BLAKE2b-256 | 0ebc60d6127dff6f50d9bd4ee133bf238422735b035fd194e3d16c62af75766e |