hdx-scraper-geonode

The HDX Scraper Geonode Library enables easy building of scrapers for extracting data from geonode servers.

Usage

The library has detailed API documentation which can be found here: http://ocha-dap.github.io/hdx-scraper-geonode/. The code for the library is here: https://github.com/ocha-dap/hdx-scraper-geonode.

GeoNodeToHDX Class

You should create an object of the GeoNodeToHDX class:

geonodetohdx = GeoNodeToHDX('https://geonode.wfp.org', downloader)
geonodetohdx = GeoNodeToHDX('https://geonode.themimu.info', downloader)

It has high level methods generate_datasets_and_showcases and delete_other_datasets:

# generate datasets and showcases reading country and layer information from the GeoNode
datasets = generate_datasets_and_showcases('maintainerid', 'orgid', 'orgname', updatefreq='Adhoc', 
                                           subnational=True)
# generate datasets and showcases reading layer information ignoring region (country) in layers call
countrydata = {'iso3': 'MMR', 'name': 'Myanmar', 'layers': None}
datasets = generate_datasets_and_showcases('maintainerid', 'orgid', 'orgname', updatefreq='Adhoc', 
                                           subnational=True, countrydata=countrydata)
# delete any datasets and associated showcases from HDX that are not in the list datasets
# (assuming matching organisation id, maintainer id and geonode url in the resource url)
delete_other_datasets(datasets)

If you need more fine grained control, it has low level methods get_locationsdata, get_layersdata, generate_dataset_and_showcase:

# get countries where count > 0
countries = geonodetohdx.get_countries(use_count=True)
# get layers for country with ISO 3 code SDN
layers = geonodetohdx.get_layers(countryiso='SDN')
# get layers for all countries
layers = get_layers(countryiso=None)

There are default terms to be ignored and mapped. These can be overridden by creating a YAML configuration with the new configuration in this format:

ignore_data:
  - deprecated

category_mapping:
  Elevation: 'elevation - topography - altitude'
  'Inland Waters': river

titleabstract_mapping:
  bridges:
    - bridges
    - transportation
    - 'facilities and infrastructure'
  idp:
    camp:
      - 'displaced persons locations - camps - shelters'
      - 'internally displaced persons - idp'
    else:
      - 'internally displaced persons - idp'

ignore_data are any terms in the abstract that mean that the dataset should not be added to HDX.

category_mapping are mappings from the category field category__gn_description to HDX metadata tags.

titleabstract_mapping are mappings from terms in the title or abstract to HDX metadata tags.

For more fine grained tuning of these, you retrieve the dictionaries and manipulate them directly:

geonodetohdx = GeoNodeToHDX('https://geonode.wfp.org', downloader)
ignore_data = geonodetohdx.get_ignore_data() 
category_mapping = geonodetohdx.get_category_mapping() 
titleabstract_mapping = geonodetohdx.get_titleabstract_mapping()

Metadata

Release files for hdx-scraper-geonode 1.1.8

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for hdx-scraper-geonode 1.1.8
File	Size	Uploaded
hdx-scraper-geonode-1.1.8.tar.gz	20.9 kB	Jan 13, 2020	Details

Built distribution (wheel)

Table of built distributions (wheels) for hdx-scraper-geonode 1.1.8
File	Interpreter	ABI	Platform	Reset
hdx_scraper_geonode-1.1.8-py2.py3-none-any.whl 10.3 kB Jan 13, 2020	Python 3, Python 2	none	any	Details

Total release size: 31.2 kB

Release files / hdx-scraper-geonode-1.1.8.tar.gz

Download URL	hdx-scraper-geonode-1.1.8.tar.gz
Size	20.9 kB
Tags	Source
SHA-256 checksum How to use checksums	`8a438b43b0262f4e596617a028789cb35819dd7ced6ebac0c4007bb73872f1d5`
BLAKE2b-256 checksum How to use checksums	`e2441914eaeedf8d8402c3fb5f627829e8500c2dced09d3d44a025a5e8144d9d`
Upload date	Jan 13, 2020
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.6.8`

Release files / hdx_scraper_geonode-1.1.8-py2.py3-none-any.whl

Download URL	hdx_scraper_geonode-1.1.8-py2.py3-none-any.whl
Size	10.3 kB
Tags	Python 2 Python 3
SHA-256 checksum How to use checksums	`a7f6b2d3b7860e97fa2868bfaeea254b1f47e18ff0a4dc764b4f223ab52f3a3b`
BLAKE2b-256 checksum How to use checksums	`1cedf91cf5d8ebab78e1e11c33e03c991b14127b40f9606bb36222a7a3d069fa`
Upload date	Jan 13, 2020
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.6.8`