Skip to main content

hydro-opendata

image

📜 中文文档

Methods and paths for obtaining, managing, and utilizing open data for hydrological scientific computations.

Background

In the era of artificial intelligence, data-driven hydrological models have been extensively researched and applied. With the advancements in remote sensing technologies and the trend towards open data sharing, accessing data has become more straightforward with a plethora of options. For researchers, questions like what data is required, what data can be accessed, where to download it, how to read it, and how to process it, are crucial. This repository aims to address these concerns.

This repository primarily focuses on external open data, categorizing data types, and creating a list. It aims to build a data flow and its tech stack that can seamlessly "download-store-process-read-write-visualize" the data.

Overall Solution

Data Framework

Main Data Sources

From our current understanding, the external data suitable for hydrological modeling includes but is not limited to:

Primary Category Secondary Category Update Frequency Data Structure Example
Basic Geography Hydrological Elements Static Vector Watershed boundary, site
Terrain Static Raster DEM, flow direction, land use
Weather & Meteorology Reanalysis Dynamic Raster ERA5
Near Real-Time Dynamic Raster GPM
Forecast Rolling Raster GFS
Imagery Satellite Remote Sensing Dynamic Raster Landsat, Sentinel, MODIS
Street View Images Static Multimedia
Surveillance Videos Dynamic Multimedia
Drone Footage Dynamic Multimedia
Crowdsourced Data POI Static Vector Baidu Map
Social Networks Dynamic Multimedia Weibo
Hydrological Data River Flow Data Dynamic Tabular GRDC

Data can be categorized based on their update frequency into static and dynamic data.

From a structural perspective, data can be classified into vector, raster, and multimedia (unstructured data).

Structure and Functional Framework

Code Repository

wis-stac

Data inventory and its metadata. Returns a data list based on AOI.

wis-downloader

Downloads data from external sources. Depending on the data source, the download methods may vary, including:

  • Integration with official APIs, e.g., bmi_era5
  • Retrieving data download links, e.g., Herbie, MultiEarth, Satpy. Most cloud data platforms like Microsoft, AWS, etc., organize data mostly as stac.

wis-processor

Preprocesses the data, such as watershed averaging, feature extraction, etc.

Uses kerchunk to convert different format data to zarr format and stores it in MinIO server. This enables cross-file reading and enhances data reading efficiency.

wis-s3api

After data processing in MinIO, it supports cross-file reading. Just provide data type, time range, and spatial range parameters to fetch the data.

For remote sensing imagery, due to the vast amount of data, it's not feasible to download and read each file. One can use stac+stackstac to directly read Sentinel or Landsat data into an xarray dataset.

wis-gistools

Integrates commonly used GIS tools, such as Kriging interpolation, Thiessen polygons, etc.

Visualization

Use leafmap to display geospatial data within the Jupyter platform.

Others

Metadata

Release files for hydro-opendata 0.0.8

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for hydro-opendata 0.0.8
File Size Uploaded Provenance Actions
hydro_opendata-0.0.8.tar.gz 43.0 kB Without provenance Details for hydro_opendata-0.0.8.tar.gz

Release files / hydro_opendata-0.0.8.tar.gz

Download URL hydro_opendata-0.0.8.tar.gz
Size 43.0 kB
Tags Source
SHA-256 checksum
How to use checksums
022da64234691c00e7ec2112f852aac01612865ef0261224f5637dd191ac2c11
BLAKE2b-256 checksum
How to use checksums
ad211402cab0051b7f1c1a8dfa549adf72529c1bd42df72b1b8d81e5a95a9fbb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.7.1 importlib_metadata/6.3.0 pkginfo/1.8.3 requests/2.28.1 requests-toolbelt/0.9.1 tqdm/4.64.0 CPython/3.9.12

Security for hydro-opendata 0.0.8

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page