geoprepare

A Python package to prepare (download, extract, process input data) for GEOCIF and related models

Free software: MIT license
Documentation: https://ritviksahajpal.github.io/geoprepare

Installation

Install from PyPI

pip install --upgrade geoprepare

Install from GitHub (development)

pip install --upgrade --no-deps --force-reinstall git+https://github.com/ritviksahajpal/geoprepare.git

Local editable install

pip install -e ".[dev]"

CDS API (for AgERA5)

If you intend to download AgERA5 data, install the CDS API by following the instructions here.

MODIS data (octvi)

Install the octvi package to download MODIS data:

pip install git+https://github.com/ritviksahajpal/octvi.git

Downloading from the NASA DAACs requires a personal app key. After installation, run octviconfig in your command prompt. Information on obtaining app keys can be found here.

Pipeline

geoprepare follows a three-stage pipeline:

Download (geodownload) - Download and preprocess global EO datasets to dir_download and dir_intermed
Extract (geoextract) - Extract EO variable statistics per admin region to dir_output
Merge (geomerge) - Merge extracted EO files into per-country/crop CSV files for ML models and AgMet graphics

All datasets store files in year-specific subfolders (e.g., dir_intermed/cpc_tmax/2024/, dir_download/nsidc/2025/).

Additional utilities:

Move (geomove) - One-time migration of existing flat directories to year-specific subfolders
Check (geocheck) - Validate that expected TIF files exist in dir_intermed after download
Diagnostics (diagnostics) - Count and summarize files in the data directories

Analysis tools (separate from the production pipeline):

Extract cells (extract_cells) - Per-cell EO timeseries dump for downstream cell-mask optimisation by geocif.cell_optimizer

Usage

config_dir = "/path/to/config"  # full path to your config directory

cfg_geoprepare = [f"{config_dir}/geobase.txt", f"{config_dir}/countries.txt", f"{config_dir}/crops.txt", f"{config_dir}/geoextract.txt"]

1. Download data (`geodownload`)

Downloads and preprocesses global EO datasets. Only requires geobase.txt. The [DATASETS] section controls which datasets are downloaded. Each dataset is processed to global 0.05° TIF files in dir_intermed.

from geoprepare import geodownload
geodownload.run([f"{config_dir}/geobase.txt"])

2. Migrate to year subfolders (`geomove`)

Moves existing files from flat directories into year-specific subfolders. Run this once after upgrading to a version with year-subfolder support. All datasets are handled: CPC, ESI, NDVI, NSIDC, CHIRPS-GEFS, LST, Soil Moisture, AgERA5, VHI, FPAR, and AEF.

from geoprepare import geomove

# Preview what would be moved (no files are changed)
geomove.run([f"{config_dir}/geobase.txt"], dry_run=True)

# Execute the migration
geomove.run([f"{config_dir}/geobase.txt"])

3. Validate downloads (`geocheck`)

Checks that all expected TIF files exist in dir_intermed and are non-empty. Writes a timestamped report to dir_logs/check/.

from geoprepare import geocheck
geocheck.run([f"{config_dir}/geobase.txt"])

4. Extract crop masks and EO data (`geoextract`)

Extracts EO variable statistics (mean, median, etc.) for each admin region, crop, and growing season.

from geoprepare import geoextract
geoextract.run(cfg_geoprepare)

5. Merge extracted data (`geomerge`)

Merges per-region/year EO CSV files into a single CSV per country-crop-season combination.

from geoprepare import geomerge
geomerge.run(cfg_geoprepare)

Optional: Per-cell EO timeseries dump (`extract_cells`)

Analysis tool, separate from the production pipeline. Produces a long-format parquet of per-cell EO values per region (one row per cell × year × doy), consumed by geocif.cell_optimizer for GA-based cell-mask optimisation. Uses the same config files as geoextract; reads its own [CELL_OPTIMIZER] section from geoextract.txt for the variable list.

from geoprepare import extract_cells
extract_cells.run(cfg_geoprepare)

Output: ${PATHS:dir_output}/cell_optimizer/{country}/{crop}/{country}_{crop}_s{season}_cells.parquet. Schema: country, region, region_id, cell_id, lat, lon, afi, year, doy, <one float32 column per configured variable>. cell_id is stable across runs as long as the AFI raster and its CRS don't change.

Config files

File	Purpose	Used by
`geobase.txt`	Paths, dataset settings, boundary file column mappings, logging	both
`countries.txt`	Per-country config (boundary files, admin levels, seasons, crops)	both
`crops.txt`	Crop masks, calendar category settings (EWCM, AMIS)	both
`geoextract.txt`	Extraction-only settings (method, threshold, parallelism)	geoprepare
`geocif.txt`	Indices/ML/agmet settings, country overrides, runtime selections	geocif

Order matters: Config files are loaded left-to-right. When the same key appears in multiple files, the last file wins. The tool-specific file (geoextract.txt or geocif.txt) must be last so its [DEFAULT] values (countries, method, etc.) override the shared defaults in countries.txt.

config_dir = "/path/to/config"

cfg_geoprepare = [f"{config_dir}/geobase.txt", f"{config_dir}/countries.txt", f"{config_dir}/crops.txt", f"{config_dir}/geoextract.txt"]
cfg_geocif = [f"{config_dir}/geobase.txt", f"{config_dir}/countries.txt", f"{config_dir}/crops.txt", f"{config_dir}/geocif.txt"]

geobase.txt

Shared paths, dataset settings, boundary file column mappings, and logging. Key sections:

[DATASETS] — Which datasets to download (e.g. ['CHIRPS', 'CPC', 'NDVI', 'ESI', 'NSIDC'])
[PATHS] — All directory paths, derived from dir_base
Per-dataset sections ([CHIRPS], [CPC], [FLDAS], etc.) — Dataset-specific settings like data URLs, variables, fill values
Boundary file sections ([adm_shapefile], [gaul1_asap_v04], etc.) — Column mappings from shapefile fields to standard names (ADM0_NAME, ADM1_NAME, ADM_ID)
[DEFAULT] — Shared defaults: start_year, end_year, parallel_process, fraction_cpus

countries.txt

Per-country configuration. Each country section specifies boundary file, admin level, seasons, crops, and EO variables. Countries are grouped by calendar category:

AMIS countries — Inherit defaults, override crops as needed
EWCM countries — Set category = EWCM, use_cropland_mask = True, custom calendar_file and boundary_file
[DEFAULT] — Shared defaults including eo_model (list of EO variables to extract)

crops.txt

Crop mask filenames (e.g. [maize] mask = Percent_Maize.tif) and calendar category settings ([EWCM], [AMIS]).

geoextract.txt

Extraction settings for geoprepare. [DEFAULT] section sets method, redo, threshold, floor/ceil, parallel_extract, countries, and forecast_seasons. Optional [CELL_OPTIMIZER] section (read by extract_cells) configures the per-cell EO timeseries dump — variables = ["ndvi", "tmax", "tmin", "precip"] for the built-in short-name mapping, or a {column: eo_var} dict for explicit control.

geocif.txt

ML and agmet settings for geocif. Contains [AGMET] plotting config, per-country crop overrides, ML model definitions, and [ML] hyperparameters.

Supported datasets

Dataset	Description	Source
AEF	AlphaEarth Foundations satellite embeddings (64-band, 10m)	source.coop
AGERA5	Agrometeorological indicators (precipitation, temperature)	CDS
AVHRR	Long-term NDVI	NOAA NCEI
CHIRPS	Rainfall estimates (v2 and v3)	CHC
CHIRPS-GEFS	15-day precipitation forecasts	CHC
CPC	Temperature (Tmax, Tmin) and precipitation	NOAA CPC
ESI	Evaporative Stress Index (4-week, 12-week)	SERVIR
FLDAS	Land surface model outputs (soil moisture, precip, temp)	NASA
FPAR	Fraction of Absorbed Photosynthetically Active Radiation	JRC
LST	Land Surface Temperature (MODIS MOD11C1)	NASA
NDVI	Vegetation index from MODIS (MOD09CMG)	NASA
NOAA S2S	Monthly-to-seasonal t2m/precip forecasts + hindcasts (4 models, leads 1-6); feeds pre-season yield forecasting incl. seasons that have not started yet	NOAA PSL
NSIDC	SMAP L4 soil moisture (surface, rootzone)	NASA NSIDC
SOIL-MOISTURE	NASA-USDA soil moisture (surface as1, subsurface as2)	NASA
VHI	Vegetation Health Index	NOAA STAR
VIIRS	Vegetation index from VIIRS (VNP09CMG)	NASA

Directory layout

All datasets organize files into year-specific subfolders. After running geomove (or on fresh downloads), the directory structure looks like:

dir_download/
  nsidc/2025/*.h5, nsidc/2026/*.h5
  chirps_gefs/2026/*.tif
  fpar/2024/*.tif, fpar/2025/*.tif
  modis_lst/*.hdf                     (flat - pymodis manages this)
  ...

dir_intermed/
  cpc_tmax/2024/*.tif, cpc_tmax/2025/*.tif
  cpc_tmin/2024/*.tif, ...
  cpc_precip/2024/*.tif, ...
  chirps/v3/global/2024/*.tif, ...    (CHIRPS already used year subfolders)
  chirps_gefs/2026/*.tif
  esi_4wk/2024/*.tif, ...
  esi_12wk/2024/*.tif, ...
  ndvi/2024/*.tif, ...
  lst/2024/*.tif, ...
  nsidc/subdaily/2025/*.tif
  nsidc/daily/surface/2025/*.tif
  nsidc/daily/rootzone/2025/*.tif
  soil_moisture_as1/2024/*.tif, ...
  soil_moisture_as2/2024/*.tif, ...
  agera5/tif/{variable}/2024/*.tif, ...
  vhi/global/2024/*.tif, ...
  aef/{country}/2018/*.tif, ..., aef/{country}/aef_avg_global.tif
  fldas/.../2024/*.tif, ...           (FLDAS already used year subfolders)

Upload package to PyPI

# 1. Bump version
uvx bump2version patch --current-version X.X.X --new-version X.X.Y pyproject.toml geoprepare/__init__.py

# 2. Clean, build, upload
rm -rf dist/ build/ *.egg-info/
uv build
uvx twine upload dist/geoprepare-X.X.Y*

Credits

This project was supported by NASA Applied Sciences Grant No. 80NSSC17K0625 through the NASA Harvest Consortium, and the NASA Acres Consortium under NASA Grant #80NSSC23M0034.

Release files for geoprepare 0.6.311

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for geoprepare 0.6.311
File	Size	Uploaded
geoprepare-0.6.311.tar.gz	14.9 MB	Sep 7, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for geoprepare 0.6.311
File	Interpreter	ABI	Platform	Reset
geoprepare-0.6.311-py3-none-any.whl 14.9 MB Sep 7, 2026	Python 3	none	any	Details

Total release size: 29.8 MB

Release files / geoprepare-0.6.311.tar.gz

Download URL	geoprepare-0.6.311.tar.gz
Size	14.9 MB
Tags	Source
SHA-256 checksum How to use checksums	`39f76a702f9e4aa94de0c1a653693838db541ef49c0e8d7bafea225b2a5ba01c`
BLAKE2b-256 checksum How to use checksums	`5a57884d629686b176d2b87536355dd3135a198e7d3da3ed8be06320f8816a0c`
Upload date	Sep 7, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.11.13`

Release files / geoprepare-0.6.311-py3-none-any.whl

Download URL	geoprepare-0.6.311-py3-none-any.whl
Size	14.9 MB
Tags	Python 3
SHA-256 checksum How to use checksums	`c777e38441992443474d329bdc673e178b89c6679e3910f39e2ee4d00a235579`
BLAKE2b-256 checksum How to use checksums	`2422123e78cdb636e4c5101a19d7965fdadbcd9b82d690416eaf2d2fa604e8ad`
Upload date	Sep 7, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.11.13`