Build Status Coverage Status

The HDX Scraper Geonode Library enables easy building of scrapers for extracting data from geonode servers.

Usage

The library has detailed API documentation which can be found here: http://ocha-dap.github.io/hdx-scraper-geonode/. The code for the library is here: https://github.com/ocha-dap/hdx-scraper-geonode.

GeoNodeToHDX Class

You should create an object of the GeoNodeToHDX class:

geonodetohdx = GeoNodeToHDX('https://geonode.wfp.org', downloader)
geonodetohdx = GeoNodeToHDX('https://geonode.themimu.info', downloader)

It has high level methods generate_datasets_and_showcases and delete_other_datasets:

# generate datasets and showcases reading country and layer information from the GeoNode
datasets = generate_datasets_and_showcases('maintainerid', 'orgid', 'orgname', updatefreq='Adhoc', 
                                           subnational=True)
# generate datasets and showcases reading layer information ignoring region (country) in layers call
countrydata = {'iso3': 'MMR', 'name': 'Myanmar', 'layers': None}
datasets = generate_datasets_and_showcases('maintainerid', 'orgid', 'orgname', updatefreq='Adhoc', 
                                           subnational=True, countrydata=countrydata)
# delete any datasets and associated showcases from HDX that are not in the list datasets
# (assuming matching organisation id, maintainer id and geonode url in the resource url)
delete_other_datasets(datasets)

If you need more fine grained control, it has low level methods get_locationsdata, get_layersdata, generate_dataset_and_showcase:

# get countries where count > 0
countries = geonodetohdx.get_countries(use_count=True)
# get layers for country with ISO 3 code SDN
layers = geonodetohdx.get_layers(countryiso='SDN')
# get layers for all countries
layers = get_layers(countryiso=None)

There are default terms to be ignored and mapped. These can be overridden by creating a YAML configuration with the new configuration in this format:

ignore_data:
  - deprecated

category_mapping:
  Elevation: 'elevation - topography - altitude'
  'Inland Waters': river

titleabstract_mapping:
  bridges:
    - bridges
    - transportation
    - 'facilities and infrastructure'
  idp:
    camp:
      - 'displaced persons locations - camps - shelters'
      - 'internally displaced persons - idp'
    else:
      - 'internally displaced persons - idp'

ignore_data are any terms in the abstract that mean that the dataset should not be added to HDX.

category_mapping are mappings from the category field category__gn_description to HDX metadata tags.

titleabstract_mapping are mappings from terms in the title or abstract to HDX metadata tags.

For more fine grained tuning of these, you retrieve the dictionaries and manipulate them directly:

geonodetohdx = GeoNodeToHDX('https://geonode.wfp.org', downloader)
ignore_data = geonodetohdx.get_ignore_data() 
category_mapping = geonodetohdx.get_category_mapping() 
titleabstract_mapping = geonodetohdx.get_titleabstract_mapping()

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.5.9)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for mcarans from gravatar.com mcarans

Credits

Author: Michael Rans

License

MIT License (MIT)

Tags

HDX scraper geonode

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

hdx-scraper-geonode-1.3.2.tar.gz (20.6 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

hdx_scraper_geonode-1.3.2-py2.py3-none-any.whl (10.0 kB view details)

Uploaded Python 2Python 3

File details

Details for the file hdx-scraper-geonode-1.3.2.tar.gz.

File metadata

  • Download URL: hdx-scraper-geonode-1.3.2.tar.gz
  • Upload date:
  • Size: 20.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.6.8

File hashes

Hashes for hdx-scraper-geonode-1.3.2.tar.gz
Algorithm Hash digest
SHA256 12664f58cbb7d232a0301023fda0769a227cb2d50ef6100b995d9f36f29cc208
MD5 17d01ebfbf69a1e82a2275914f91787b
BLAKE2b-256 715eb3bded620fc6fdd0e78618161323c01236fba52e747daa94b1a56d1af023

See more details on using hashes here.

File details

Details for the file hdx_scraper_geonode-1.3.2-py2.py3-none-any.whl.

File metadata

  • Download URL: hdx_scraper_geonode-1.3.2-py2.py3-none-any.whl
  • Upload date:
  • Size: 10.0 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.6.8

File hashes

Hashes for hdx_scraper_geonode-1.3.2-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 d6030d220ff043ad1e64967c5f3700c9dc8f736629a9443d4daff84a7157cc6c
MD5 6eab4f91bea285240494d07f0e54db2f
BLAKE2b-256 6e9b1a5790ff9848f083a55790d6881450079f8bd7267ec9839213b87279e519

See more details on using hashes here.

Release history Release notifications | RSS feed

1.5.9

2 files

1.5.8

2 files

1.5.7

2 files

1.5.6

2 files

1.5.5

2 files

1.5.4

2 files

1.5.3

2 files

1.5.2

2 files

1.5.1

2 files

1.5.0

2 files

1.4.9

1 file

1.4.8

1 file

1.4.7

1 file

1.4.6

1 file

1.4.5

1 file

1.4.4

1 file

1.4.3

1 file

1.4.2

1 file

1.4.1

2 files

1.4.0

2 files

1.3.6

2 files

1.3.5

2 files

1.3.4

2 files

1.3.3

2 files

This release

1.3.2 This release

2 files

1.3.1

2 files

1.3.0

2 files

1.2.9

2 files

1.2.8

2 files

1.2.7

2 files

1.2.6

2 files

1.2.5

2 files

1.2.4

2 files

1.2.3

2 files

1.2.2

2 files

1.2.1

2 files

1.2.0

2 files

1.1.9

2 files

1.1.8

2 files

1.1.7

2 files

1.1.6

2 files

1.1.5

2 files

1.1.4

2 files

1.1.3

2 files

1.1.2

2 files

1.1.1

2 files

1.1.0

2 files

1.0.9

2 files

1.0.8

2 files

1.0.7

2 files

1.0.6

2 files

1.0.5

2 files

1.0.4

2 files

1.0.3

2 files

1.0.2

2 files

1.0.1

2 files

1.0.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page