Skip to main content

Harvester Next Generation Core for CKAN

Project description

Harvester Next Generation for CKAN

Install

pip install ckan-harvester

Use data.json sources

from harvester.data_json import DataJSON
dj = DataJSON()
dj.url = 'https://data.iowa.gov/data.json'
ret, error = dj.download_data_json()
print(ret, error)
# True None

ret, error = dj.load_data_json()
print(ret, error)
# True None

ret, errors = dj.validate_json()
print(ret, errors)
# False ['Error validating JsonSchema: \'bureauCode\' is a required property ...

# full dict with the source
print(dj.data_json)
"""
{
	'@context': 'https://project-open-data.cio.gov/v1.1/schema/catalog.jsonld',
	'@id': 'https://data.iowa.gov/data.json',
	'@type': 'dcat:Catalog',
	'conformsTo': 'https://project-open-data.cio.gov/v1.1/schema',
	'describedBy': 'https://project-open-data.cio.gov/v1.1/schema/catalog.json',
	'dataset': [{
		'accessLevel': 'public',
		'landingPage': 'https://data.iowa.gov/d/23jk-3uwr',
		'issued': '2017-01-30',
		'@type': 'dcat:Dataset',

        ... 
"""
# just headers
print(dj.headers)

"""
{
'@context': 'https://project-open-data.cio.gov/v1.1/schema/catalog.jsonld',
'@id': 'https://data.iowa.gov/data.json',
'@type': 'dcat:Catalog',
'conformsTo': 'https://project-open-data.cio.gov/v1.1/schema',
'describedBy': 'https://project-open-data.cio.gov/v1.1/schema/catalog.json',
}
"""

for dataset in dj.datasets:
    print(dataset['title'])

Impaired Streams 2014
2009-2010 Iowa Public School District Boundaries
2015 - 2016 Iowa Public School District Boundaries
Impaired Streams 2010
Impaired Lakes 2014
2007-2008 Iowa Public School District Boundaries
Impaired Streams 2012
2011-2012 Iowa Public School District Boundaries
Active and Completed Watershed Projects - IDALS
2012-2013 Iowa Public School District Boundaries
2010-2011 Iowa Public School District Boundaries
2016-2017 Iowa Public School District Boundaries
2014 - 2015 Iowa Public School District Boundaries
Impaired Lakes 2008
2008-2009 Iowa Public School District Boundaries
2013-2014 Iowa Public School District Boundaries
Impaired Lakes 2010
Impaired Lakes 2012
Impaired Streams 2008

Use CSW sources

from harvester.csw import CSWSource
c = CSWSource(url='http://data.nconemap.com/geoportal/csw?Request=GetCapabilities&Service=CSW&Version=2.0.2')
csw.connect_csw()
 # True

csw_info = csw.read_csw_info()
print('CSW title: {}'.format(csw_info['identification']['title']))
 # CSW title: ArcGIS Server Geoportal Extension 10 - OGC CSW 2.0.2 ISO AP

Development

To setup a develop environment, clone the repository and in a virtualenv install the dependencies

pip install -r requirements.txt

This will install the library in development mode, and other libraries for tests.

Test

Then to run the test suite with pytest:

pytest

We use pytest-vcr based on the wonderful VCRpy, to mock http requests. In this way, we don't need to hit the real internet to run our test (which is very fragile and slow), because there is a mocked version of a each response needed by tests, in vcr's cassettes format.

In order to update these cassettes just run as following:

pytest --vcr-record=all

To actually hit the internet without use mocks, disable the plugin

pytest --disable-vcr

Tests without a CKAN instance

python -m pytest tests

================ test session starts =============
platform linux -- Python 3.6.8, pytest-5.2.0, py-1.8.0, pluggy-0.13.0
rootdir: /home/hudson/dev/datopian/ckan-ng-harvester-core
plugins: vcr-1.0.2
collected 17 items                                                                                                                                                          

tests/test_csw_dataset_adapter.py ....      [ 23%]
tests/test_data_json.py .......             [ 64%]
tests/test_datajson_dataset_adapter.py .....[100%]

=============== 17 passed in 17.52s ==============

Tests with a CKAN instance.
You will need to copy settings.py file to local_settings.py file and fill the required values.
You can use a local or remote CKAN instance.

python -m pytest tests_with_ckan/test_harvest.py

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distribution

ckan_harvester-0.109-py3-none-any.whl (46.8 kB view details)

Uploaded Python 3

File details

Details for the file ckan_harvester-0.109-py3-none-any.whl.

File metadata

  • Download URL: ckan_harvester-0.109-py3-none-any.whl
  • Upload date:
  • Size: 46.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.15.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.6.8

File hashes

Hashes for ckan_harvester-0.109-py3-none-any.whl
Algorithm Hash digest
SHA256 6a98ff4316d8a649e233164941226945f1f4d36811517db7c846136db93df685
MD5 81e25359358edd38a4568ac16a456e1e
BLAKE2b-256 0ebc60d6127dff6f50d9bd4ee133bf238422735b035fd194e3d16c62af75766e

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page