The HDX Scraper Geonode Library enables easy building of scrapers for extracting data from geonode servers.
Usage
The library has detailed API documentation which can be found here: http://ocha-dap.github.io/hdx-scraper-geonode/. The code for the library is here: https://github.com/ocha-dap/hdx-scraper-geonode.
GeoNodeToHDX Class
You should create an object of the GeoNodeToHDX class which has methods: get_locationsdata, get_layersdata, generate_dataset_and_showcase.
geonodetohdx = GeoNodeToHDX('https://geonode.wfp.org', downloader)
# get countries where count > 0
countries = geonodetohdx.get_countries(use_count=True)
# get layers for country with ISO 3 code SDN
layers = geonodetohdx.get_layers(countryiso='SDN')
# get layers for all countries
geonodetohdx = GeoNodeToHDX('https://geonode.themimu.info', downloader)
layers = get_layers(countryiso=None)
There are default terms to be ignored and mapped. These can be overridden by creating a YAML configuration with the new configuration in this format:
ignore_data:
- deprecated
category_mapping:
Elevation: 'elevation - topography - altitude'
'Inland Waters': river
titleabstract_mapping:
bridges:
- bridges
- transportation
- 'facilities and infrastructure'
idp:
camp:
- 'displaced persons locations - camps - shelters'
- 'internally displaced persons - idp'
else:
- 'internally displaced persons - idp'
ignore_data are any terms in the abstract that mean that the dataset should not be added to HDX.
category_mapping are mappings from the category field category__gn_description to HDX metadata tags.
titleabstract_mapping are mappings from terms in the title or abstract to HDX metadata tags.
For more fine grained tuning of these, you retrieve the dictionaries and manipulate them directly:
geonodetohdx = GeoNodeToHDX('https://geonode.wfp.org', downloader)
ignore_data = geonodetohdx.get_ignore_data()
category_mapping = geonodetohdx.get_category_mapping()
titleabstract_mapping = geonodetohdx.get_titleabstract_mapping()
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file hdx-scraper-geonode-1.0.1.tar.gz.
File metadata
- Download URL: hdx-scraper-geonode-1.0.1.tar.gz
- Upload date:
- Size: 16.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.6.8
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
b0bfebd54032c1fbf68b58bd89ec886e6e88b583b70d4c08037d34a3044715da
|
|
| MD5 |
233dc2075ad047d80c2172e5388dbf9a
|
|
| BLAKE2b-256 |
8c867d2a99d240d5c1e2d2864645f5fda14a040508d197aa06a63eac6fa9737b
|
File details
Details for the file hdx_scraper_geonode-1.0.1-py2.py3-none-any.whl.
File metadata
- Download URL: hdx_scraper_geonode-1.0.1-py2.py3-none-any.whl
- Upload date:
- Size: 8.1 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.6.8
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
f3a7837a6c2da3c0085ea722302e393a77827b96e37038f7b928df6a04b99af0
|
|
| MD5 |
a7ed8996740ad11c34bd3e9a1f4cd4e8
|
|
| BLAKE2b-256 |
b8ee1982f6b2847c37aaf201aeb9eb5320f0e66ed102a40be473770791a25bda
|
