geoextract_gee

Google Earth Engine backend extension for geoprepare (geoextract).

This package provides an optional Google Earth Engine (GEE) execution path for Earth Observation (EO) data extraction. It orchestrates task combinations, processes them using GEE, polls the tasks, and downloads the resulting CSVs from Google Cloud Storage (GCS).

Prerequisites

• Google Earth Engine Account: Ensure you have access and a project set up.
• Google Cloud Storage: A GCS bucket is required to export the results before downloading.
• Environment: It is recommended to refer to installer for setting up the environment required for geocif and geoprepare.

Installation

pip install geoextract-gee

This also installs geoprepare, which it extends. geoprepare switches to this backend when use_gee = True (see Configuration).

Authentication

Set key in [GEE] to a service account JSON key, as a path relative to geoprepare's metadata folder. Without it (or with key = none), the standard Earth Engine sign-in (ee.Authenticate()) runs the first time; follow its instructions.

Configuration

Turn the backend on in [DEFAULT] of your geoprepare config, and put its settings in a [GEE] section:

[DEFAULT]
use_gee = True
redo = False

[GEE]
project = your-gcp-project-name
bucket = your-gcs-bucket-name

# Optional
# Service account key, relative to the metadata folder. Default: interactive sign-in.
key = gee_key.json
# Compute extra counts and sums per region, for checking results. Costs more EECU. Default False.
audit_stats = False
# One GEE task per country instead of one per region (see Architecture). Default False.
parallel_regions = True
# Price of one EECU-hour, in dollars, for the cost shown while polling. Default 0.4.
eecu_h_pricing = 0.4
# Extract variables that have no GEE handler with the local tiles backend
# (see "Variables not in Earth Engine" below). Default False.
fallback_local = False
# Write GEE outputs to dir_output/gee/ instead of dir_output/, for comparing the
# two backends. A debugging aid; geomerge reads dir_output. Default False.
use_subdir = False
# Label for every Earth Engine task, so its compute shows per label in Cloud billing
# and monitoring. 1-63 characters. Default: geoprepare's [DEFAULT] project_name.
workload_tag = my-project

Exports go to gs://<bucket>/gee_extract/<project_name>/, where project_name is geoprepare's [DEFAULT] project_name. Give each setup that shares a bucket its own project_name: two setups with the same name and bucket write to the same folder and reuse each other's Earth Engine tasks.

Configs from before version 0.2.0 have these settings in [DEFAULT] with a gee_ prefix (gee_project, gee_bucket, gee_key, gee_audit_stats, gee_parallel_regions, gee_eecu_h_pricing, gee_fallback_local) and use_gee_subdir. They still work, with a warning naming the new setting.

Supported Datasets

The GEE backend currently supports extraction for the following geoprepare datasets:

Variables not in Earth Engine

Some geoprepare variables have no copy in Earth Engine or its community catalog, so this backend cannot extract them:

Other variables without a handler (for example lst, fpar, ndvi_cdr) are treated the same way.

By default (fallback_local = False) the run stops with an error listing these variables, before any GEE task is submitted. With fallback_local = True, each one is logged with a warning and extracted by geoprepare's local tiles backend while the GEE tasks run. The local backend reads files that geodownload has already written to dir_intermed; geoextract does not download them. If they are missing, the run stops before submitting any task and names the geodownload dataset to run (for example CHIRTS-ERA5). Fallback variables are written to dir_output, not to dir_output/gee/, even with use_subdir = True. They are extracted one combination at a time.

Architecture

1. Task Submission: The pipeline extracts data using one of two modes, controlled by parallel_regions in [GEE]:
   • Sequential Mode (parallel_regions = False): For each region and variable combination, a separate GEE task is created.
   • Parallel Batch Mode (parallel_regions = True): All regions within a country are bundled into a single ee.FeatureCollection and processed simultaneously by GEE via reduceRegion mapping. This submits a single "bulk" GEE task per year/variable, reducing the task queue size and may also avoid quota limits.
     1. Note: For variables requiring heavy reductions (e.g., nsidc converting 3-hourly images to daily means), Parallel Batch Mode may cause GEE tasks to take more time to run, time out or run out of memory. In case of error, using Sequential Mode (parallel_regions = False) is recommended for these variables.
     2. Note: In Parallel Batch Mode, incremental updates for the current year do not generate partial files like Sequential Mode. Instead, the pipeline automatically re-extracts the entire year and overwrites the local CSVs to ensure seamless data coverage without complex local file merging and different time windows required by each variable.
2. Polling: The process tracks the submitted tasks and polls Earth Engine until all tasks are marked as COMPLETED or FAILED. To prevent unexpected billing, polling monitors EECU usage and cancels runaway tasks exceeding defined limits.
3. Download and Format: Once tasks are complete, the resulting CSVs are downloaded concurrently from Google Cloud Storage using transfer_manager. If Parallel Batch Mode was used, the bulk CSV is automatically parsed and split locally into the individual region CSVs expected by geomerge.