Build Status Coverage Status

The HDX Scraper Geonode Library enables easy building of scrapers for extracting data from geonode servers.

Usage

The library has detailed API documentation which can be found here: http://ocha-dap.github.io/hdx-scraper-geonode/. The code for the library is here: https://github.com/ocha-dap/hdx-scraper-geonode.

GeoNodeToHDX Class

You should create an object of the GeoNodeToHDX class which has methods: get_locationsdata, get_layersdata, generate_dataset_and_showcase.

geonodetohdx = GeoNodeToHDX('https://geonode.wfp.org', downloader)
# get countries where count > 0
countries = geonodetohdx.get_countries(use_count=True)
# get layers for country with ISO 3 code SDN
layers = geonodetohdx.get_layers(countryiso='SDN')
# get layers for all countries
geonodetohdx = GeoNodeToHDX('https://geonode.themimu.info', downloader)
layers = get_layers(countryiso=None)

There are default terms to be ignored and mapped. These can be overridden by creating a YAML configuration with the new configuration in this format:

ignore_data:
  - deprecated

category_mapping:
  Elevation: 'elevation - topography - altitude'
  'Inland Waters': river

titleabstract_mapping:
  bridges:
    - bridges
    - transportation
    - 'facilities and infrastructure'
  idp:
    camp:
      - 'displaced persons locations - camps - shelters'
      - 'internally displaced persons - idp'
    else:
      - 'internally displaced persons - idp'

ignore_data are any terms in the abstract that mean that the dataset should not be added to HDX.

category_mapping are mappings from the category field category__gn_description to HDX metadata tags.

titleabstract_mapping are mappings from terms in the title or abstract to HDX metadata tags.

For more fine grained tuning of these, you retrieve the dictionaries and manipulate them directly:

geonodetohdx = GeoNodeToHDX('https://geonode.wfp.org', downloader)
ignore_data = geonodetohdx.get_ignore_data() 
category_mapping = geonodetohdx.get_category_mapping() 
titleabstract_mapping = geonodetohdx.get_titleabstract_mapping()

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.5.9)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for mcarans from gravatar.com mcarans

Credits

Author: Michael Rans

License

MIT License (MIT)

Tags

HDX scraper geonode

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

hdx-scraper-geonode-1.0.8.tar.gz (18.2 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

hdx_scraper_geonode-1.0.8-py2.py3-none-any.whl (8.8 kB view details)

Uploaded Python 2Python 3

File details

Details for the file hdx-scraper-geonode-1.0.8.tar.gz.

File metadata

  • Download URL: hdx-scraper-geonode-1.0.8.tar.gz
  • Upload date:
  • Size: 18.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.6.8

File hashes

Hashes for hdx-scraper-geonode-1.0.8.tar.gz
Algorithm Hash digest
SHA256 e799153886653e6cb561d4cda42d6131dd82c70524967412649c70813d2cd3e1
MD5 535f8d7c0357d48e22e502c579c9d66b
BLAKE2b-256 4fa8a62c8c399b9542fa0d89543c8fdf685f61d54ee8a1335a7a1ecab29a94dc

See more details on using hashes here.

File details

Details for the file hdx_scraper_geonode-1.0.8-py2.py3-none-any.whl.

File metadata

  • Download URL: hdx_scraper_geonode-1.0.8-py2.py3-none-any.whl
  • Upload date:
  • Size: 8.8 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.6.8

File hashes

Hashes for hdx_scraper_geonode-1.0.8-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 6c153421d38bc2b693dc0d0a5182c5fdb55b1f1767bb602425692e353be25043
MD5 b286de1896e6abdf0c891cf8c9aa645e
BLAKE2b-256 0d0bfb1858187b11d20ee9bbcd5fa5265b05387b2b9343ad1439e2210cc248f2

See more details on using hashes here.

Release history Release notifications | RSS feed

1.5.9

2 files

1.5.8

2 files

1.5.7

2 files

1.5.6

2 files

1.5.5

2 files

1.5.4

2 files

1.5.3

2 files

1.5.2

2 files

1.5.1

2 files

1.5.0

2 files

1.4.9

1 file

1.4.8

1 file

1.4.7

1 file

1.4.6

1 file

1.4.5

1 file

1.4.4

1 file

1.4.3

1 file

1.4.2

1 file

1.4.1

2 files

1.4.0

2 files

1.3.6

2 files

1.3.5

2 files

1.3.4

2 files

1.3.3

2 files

1.3.2

2 files

1.3.1

2 files

1.3.0

2 files

1.2.9

2 files

1.2.8

2 files

1.2.7

2 files

1.2.6

2 files

1.2.5

2 files

1.2.4

2 files

1.2.3

2 files

1.2.2

2 files

1.2.1

2 files

1.2.0

2 files

1.1.9

2 files

1.1.8

2 files

1.1.7

2 files

1.1.6

2 files

1.1.5

2 files

1.1.4

2 files

1.1.3

2 files

1.1.2

2 files

1.1.1

2 files

1.1.0

2 files

1.0.9

2 files

This release

1.0.8 This release

2 files

1.0.7

2 files

1.0.6

2 files

1.0.5

2 files

1.0.4

2 files

1.0.3

2 files

1.0.2

2 files

1.0.1

2 files

1.0.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page